2020-07-17T11:41:36.314 INFO:root:teuthology version: 1.0.0-10de4c5 2020-07-17T11:41:36.317 DEBUG:teuthology.run:Teuthology command: teuthology --block --owner scheduled_dang@sidious.arb.redhat.com --verbose --archive /home/teuthworker/archive/dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi/5232112 --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --name dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi --lock -- /tmp/teuthology-worker.jpa2ila3.tmp 2020-07-17T11:41:36.348 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-07-17T11:41:36.417 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi/5232112 branch: wip-dang-zipper-6 description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: null first_in_suite: false job_id: '5232112' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-dang-zipper-6 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) sha1: 4dfb1075abb69d7743ad054aace9aa373509ed36 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4dfb1075abb69d7743ad054aace9aa373509ed36 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-dang-zipper-6 sha1: 4dfb1075abb69d7743ad054aace9aa373509ed36 owner: scheduled_dang@sidious.arb.redhat.com priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 4dfb1075abb69d7743ad054aace9aa373509ed36 sleep_before_teardown: 0 suite: rgw suite_branch: wip-dang-zipper-6 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-zipper-6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 4dfb1075abb69d7743ad054aace9aa373509ed36 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.8462 2020-07-17T11:41:36.418 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-zipper-6/qa; will attempt to use it 2020-07-17T11:41:36.419 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-zipper-6/qa/tasks 2020-07-17T11:41:36.419 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-07-17T11:41:36.421 INFO:teuthology.task.internal:Checking packages... 2020-07-17T11:41:36.447 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '4dfb1075abb69d7743ad054aace9aa373509ed36' 2020-07-17T11:41:36.447 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-07-17T11:41:36.448 INFO:teuthology.packaging:ref: None 2020-07-17T11:41:36.448 INFO:teuthology.packaging:tag: None 2020-07-17T11:41:36.449 INFO:teuthology.packaging:branch: wip-dang-zipper-6 2020-07-17T11:41:36.449 INFO:teuthology.packaging:sha1: 4dfb1075abb69d7743ad054aace9aa373509ed36 2020-07-17T11:41:36.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-dang-zipper-6 2020-07-17T11:41:36.635 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-3281-g4dfb1075abb-1bionic 2020-07-17T11:41:36.635 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-07-17T11:41:36.646 INFO:teuthology.task.internal:no buildpackages task found 2020-07-17T11:41:36.646 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-07-17T11:41:36.696 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-07-17T11:41:36.697 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-07-17T11:41:36.757 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:41:46.774 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:41:56.790 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:42:06.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:42:16.828 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:42:26.849 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:42:36.868 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:42:46.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:42:56.903 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:43:06.922 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:43:16.942 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:43:26.962 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:43:36.985 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:43:47.004 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:43:57.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:44:07.038 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:44:17.057 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:44:27.083 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:44:37.099 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:44:47.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:44:57.136 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:45:07.155 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:45:17.176 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:45:27.197 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:45:37.261 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:45:47.280 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:45:57.305 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:46:07.333 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:46:17.355 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:46:27.374 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:46:37.393 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:46:47.417 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:46:57.448 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:47:07.468 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:47:17.486 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:47:27.570 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:47:37.590 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:47:47.612 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:47:57.624 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:48:07.642 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:48:17.659 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:48:27.676 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:48:37.698 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:48:47.721 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:48:57.758 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:49:07.777 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:49:17.793 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:49:27.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:49:37.841 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:49:47.864 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:49:57.880 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:50:07.898 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:50:17.917 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:50:27.940 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:50:37.956 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:50:47.978 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:50:57.997 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:51:08.019 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:51:18.036 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:51:28.057 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:51:38.083 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:51:48.100 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-07-17T11:51:58.121 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-07-17T11:52:08.140 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_dang@sidious.arb.redhat.com', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi/5232112'} 2020-07-17T11:52:08.167 DEBUG:teuthology.lock.ops:locked ubuntu@smithi146.front.sepia.ceph.com, ubuntu@smithi052.front.sepia.ceph.com 2020-07-17T11:52:08.250 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f02e13e8c80>, signals=[15]) 2020-07-17T11:52:08.457 INFO:teuthology.provision.fog.smithi146:Scheduling deploy of ubuntu 18.04 2020-07-17T11:52:08.462 INFO:teuthology.provision.fog.smithi052:Scheduling deploy of ubuntu 18.04 2020-07-17T11:52:08.801 INFO:teuthology.orchestra.console:Power off smithi052 2020-07-17T11:52:08.803 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-07-17T11:52:08.826 INFO:teuthology.orchestra.console:Power off smithi146 2020-07-17T11:52:08.827 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-07-17T11:52:08.840 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:08.857 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:12.882 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:12.895 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:17.000 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:17.019 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:17.215 INFO:teuthology.orchestra.console:Power off for smithi052 completed 2020-07-17T11:52:17.317 INFO:teuthology.orchestra.console:Power on smithi052 2020-07-17T11:52:17.318 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-07-17T11:52:17.346 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:21.138 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:21.365 INFO:teuthology.orchestra.console:Power off for smithi146 completed 2020-07-17T11:52:21.372 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:21.466 INFO:teuthology.orchestra.console:Power on smithi146 2020-07-17T11:52:21.467 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-07-17T11:52:21.490 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:25.487 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:25.526 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:25.729 INFO:teuthology.orchestra.console:Power on for smithi052 completed 2020-07-17T11:52:25.831 INFO:teuthology.provision.fog.smithi052:Waiting for deploy to finish 2020-07-17T11:52:29.642 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-17T11:52:29.856 INFO:teuthology.orchestra.console:Power on for smithi146 completed 2020-07-17T11:52:29.958 INFO:teuthology.provision.fog.smithi146:Waiting for deploy to finish 2020-07-17T11:55:12.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:55:21.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:55:30.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:55:31.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:55:39.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:55:40.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:55:48.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:55:49.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:55:58.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:07.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:16.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:25.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:26.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:26.070 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-07-17T11:56:32.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:32.086 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-07-17T11:56:34.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:38.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:44.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:44.134 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-07-17T11:56:50.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:56:50.249 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-07-17T11:56:50.253 INFO:teuthology.orchestra.run.smithi052:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-07-17T11:56:50.635 INFO:teuthology.provision.fog.smithi052:Node is ready 2020-07-17T11:56:50.636 INFO:teuthology.orchestra.run.smithi052:> hostname 2020-07-17T11:56:50.683 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052.front.sepia.ceph.com 2020-07-17T11:56:50.684 INFO:teuthology.orchestra.run.smithi052:> grep smithi052.front.sepia.ceph.com /etc/hosts 2020-07-17T11:56:50.731 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52 smithi052.front.sepia.ceph.com smithi052 2020-07-17T11:56:50.732 INFO:teuthology.orchestra.run.smithi052:> sudo hostname smithi052 2020-07-17T11:56:50.796 INFO:teuthology.orchestra.run.smithi052:> sudo sed -i -e 's/smithi052.front.sepia.ceph.com/smithi052/g' /etc/hosts 2020-07-17T11:56:50.853 INFO:teuthology.orchestra.run.smithi052:> sudo sed -i -e 's/172.21.15.52/172.21.15.52/g' /etc/hosts 2020-07-17T11:56:50.914 INFO:teuthology.orchestra.run.smithi052:> sudo sed -i -e 's/smithi052.front.sepia.ceph.com/smithi052/g' /etc/hostname 2020-07-17T11:56:50.972 INFO:teuthology.orchestra.run.smithi052:> sudo hostname smithi052 2020-07-17T11:56:51.027 INFO:teuthology.provision.fog.smithi052:Deploy complete! 2020-07-17T11:56:51.028 DEBUG:teuthology.parallel:result is None 2020-07-17T11:57:40.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:57:40.816 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-07-17T11:57:40.817 INFO:teuthology.orchestra.run.smithi146:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-07-17T11:57:41.169 INFO:teuthology.provision.fog.smithi146:Node is ready 2020-07-17T11:57:41.169 INFO:teuthology.orchestra.run.smithi146:> hostname 2020-07-17T11:57:41.216 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146.front.sepia.ceph.com 2020-07-17T11:57:41.217 INFO:teuthology.orchestra.run.smithi146:> grep smithi146.front.sepia.ceph.com /etc/hosts 2020-07-17T11:57:41.265 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146 smithi146.front.sepia.ceph.com smithi146 2020-07-17T11:57:41.266 INFO:teuthology.orchestra.run.smithi146:> sudo hostname smithi146 2020-07-17T11:57:41.320 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/smithi146.front.sepia.ceph.com/smithi146/g' /etc/hosts 2020-07-17T11:57:41.383 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/172.21.15.146/172.21.15.146/g' /etc/hosts 2020-07-17T11:57:41.443 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/smithi146.front.sepia.ceph.com/smithi146/g' /etc/hostname 2020-07-17T11:57:41.498 INFO:teuthology.orchestra.run.smithi146:> sudo hostname smithi146 2020-07-17T11:57:41.553 INFO:teuthology.provision.fog.smithi146:Deploy complete! 2020-07-17T11:57:41.553 DEBUG:teuthology.parallel:result is None 2020-07-17T11:57:41.733 INFO:teuthology.lock.ops:Checking smithi146.front.sepia.ceph.com 2020-07-17T11:57:41.735 INFO:teuthology.lock.ops:New key found. Updating... 2020-07-17T11:57:41.764 INFO:teuthology.lock.ops:Checking smithi052.front.sepia.ceph.com 2020-07-17T11:57:41.765 INFO:teuthology.lock.ops:New key found. Updating... 2020-07-17T11:57:41.790 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: set os type and version on server 2020-07-17T11:57:41.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-17T11:57:41.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:57:41.924 INFO:teuthology.orchestra.run.smithi146:> true 2020-07-17T11:57:42.123 INFO:teuthology.orchestra.run.smithi146:> uname -m 2020-07-17T11:57:42.173 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2020-07-17T11:57:42.174 INFO:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2020-07-17T11:57:42.221 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu" 2020-07-17T11:57:42.221 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-17T11:57:42.221 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu 2020-07-17T11:57:42.221 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian 2020-07-17T11:57:42.222 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-17T11:57:42.222 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="18.04" 2020-07-17T11:57:42.222 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-17T11:57:42.222 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-17T11:57:42.222 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-17T11:57:42.223 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-17T11:57:42.223 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=bionic 2020-07-17T11:57:42.223 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=bionic 2020-07-17T11:57:42.225 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2020-07-17T11:57:42.250 INFO:teuthology.lock.ops:Updating [smithi052.front.sepia.ceph.com]: set os type and version on server 2020-07-17T11:57:42.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-17T11:57:42.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:57:42.327 INFO:teuthology.orchestra.run.smithi052:> true 2020-07-17T11:57:42.531 INFO:teuthology.orchestra.run.smithi052:> uname -m 2020-07-17T11:57:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2020-07-17T11:57:42.581 INFO:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2020-07-17T11:57:42.628 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="Ubuntu" 2020-07-17T11:57:42.628 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-17T11:57:42.628 INFO:teuthology.orchestra.run.smithi052.stdout:ID=ubuntu 2020-07-17T11:57:42.629 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE=debian 2020-07-17T11:57:42.629 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-17T11:57:42.629 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="18.04" 2020-07-17T11:57:42.629 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-17T11:57:42.629 INFO:teuthology.orchestra.run.smithi052.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-17T11:57:42.630 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-17T11:57:42.630 INFO:teuthology.orchestra.run.smithi052.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-17T11:57:42.630 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_CODENAME=bionic 2020-07-17T11:57:42.630 INFO:teuthology.orchestra.run.smithi052.stdout:UBUNTU_CODENAME=bionic 2020-07-17T11:57:42.631 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2020-07-17T11:57:42.651 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-07-17T11:57:42.688 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRym4vepUOl2/NXzCRLsZq2b/47oucdP+ocwQB0Bz64k/bJ0y8hUy3VGEfznebhWoaFcPdamYppBuqJ7dE+6H7xtNrKu+Vbp5WEGRFikdeCoLSb6iw0kbad+n82YKvd/K78sBc6uCstOykJumi82MlspRWXmUe873zCDXoGbQ/QAS/JCJ7a9FSyYL8zYkWZ2MueD01C0C3AGdrkecilm16odKiOnUhiU7Hi8+b5FY46K/FN4nKLYoTLXRWAlJRaOa9uc/lR1jwYb9UzewcyVNKLGLGG01TTvLkU7n04vqSfzxvrd5qzbtX6eseJjrxQeUGMsNwpxeHhRY+wSmkLBaD smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCloEPXlv7laKq901S6qmHpGoaIMf1JrkGkSkzwRt3h1ROyX2Ct+bNpFeYY5wSXYHvAQpBj1UDv1akRuhp1ni69yiqm2fR5a8bOhfMPmVW4QXnLvOf1sD1gKiGBtXfdxqQqyM42035eYHaHay9laARJhsEQaNXuLUwL5AgSgUpxrzcN3vH6QryI0Cif3iCZwshjs2yG4pTBUwPcxrrh2W+xiV+YCJBwYsPMnUf6/9eZ0OFlbIva/mqMCjqF4EeJARYn4oqwwbdEuJBP9NGtNdvajktdr0wJoAXVRE+RDXtEiT7PHr5AzwNGq7ADl/s4HWZg4mUsSkuYzvcA+fHigPop 2020-07-17T11:57:42.689 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-07-17T11:57:42.754 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-07-17T11:57:42.760 INFO:teuthology.task.internal:Saving configuration 2020-07-17T11:57:42.786 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-07-17T11:57:42.794 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-07-17T11:57:42.813 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-07-17 11:52:08.145938', 'description': '/home/teuthworker/archive/dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi/5232112', 'locked_by': 'scheduled_dang@sidious.arb.redhat.com', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC7O9B4lwSDvU5IGLitEWytELW+Hxp6Epm/9Rha3F9Uv', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi146.front.sepia.ceph.com'} 2020-07-17T11:57:42.833 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-07-17 11:52:08.145958', 'description': '/home/teuthworker/archive/dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi/5232112', 'locked_by': 'scheduled_dang@sidious.arb.redhat.com', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILwJtDzbHAGV4vpmTU/AGPmDgz/r+9PEHPUhU8Qg4LX/', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi052.front.sepia.ceph.com'} 2020-07-17T11:57:42.834 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-07-17T11:57:42.846 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-07-17T11:57:42.847 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-07-17T11:57:42.847 INFO:teuthology.run_tasks:Running task console_log... 2020-07-17T11:57:42.930 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f02e131d620>, signals=[15]) 2020-07-17T11:57:42.931 INFO:teuthology.run_tasks:Running task internal.connect... 2020-07-17T11:57:42.941 INFO:teuthology.task.internal:Opening connections... 2020-07-17T11:57:42.942 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2020-07-17T11:57:42.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:57:43.071 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2020-07-17T11:57:43.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:57:43.147 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-07-17T11:57:43.161 INFO:teuthology.orchestra.run.smithi146:> uname -m 2020-07-17T11:57:43.216 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2020-07-17T11:57:43.217 INFO:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2020-07-17T11:57:43.268 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu" 2020-07-17T11:57:43.268 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-17T11:57:43.269 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu 2020-07-17T11:57:43.269 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian 2020-07-17T11:57:43.269 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-17T11:57:43.269 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="18.04" 2020-07-17T11:57:43.269 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-17T11:57:43.270 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-17T11:57:43.270 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-17T11:57:43.270 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-17T11:57:43.270 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=bionic 2020-07-17T11:57:43.270 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=bionic 2020-07-17T11:57:43.271 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2020-07-17T11:57:43.298 INFO:teuthology.orchestra.run.smithi052:> uname -m 2020-07-17T11:57:43.308 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2020-07-17T11:57:43.309 INFO:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2020-07-17T11:57:43.355 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="Ubuntu" 2020-07-17T11:57:43.356 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-17T11:57:43.356 INFO:teuthology.orchestra.run.smithi052.stdout:ID=ubuntu 2020-07-17T11:57:43.356 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE=debian 2020-07-17T11:57:43.356 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-17T11:57:43.357 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="18.04" 2020-07-17T11:57:43.357 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-17T11:57:43.357 INFO:teuthology.orchestra.run.smithi052.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-17T11:57:43.357 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-17T11:57:43.357 INFO:teuthology.orchestra.run.smithi052.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-17T11:57:43.358 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_CODENAME=bionic 2020-07-17T11:57:43.358 INFO:teuthology.orchestra.run.smithi052.stdout:UBUNTU_CODENAME=bionic 2020-07-17T11:57:43.358 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2020-07-17T11:57:43.378 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-07-17T11:57:43.415 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-07-17T11:57:43.423 INFO:teuthology.task.internal:Checking for old test directory... 2020-07-17T11:57:43.423 INFO:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2020-07-17T11:57:43.426 INFO:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2020-07-17T11:57:43.431 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-07-17T11:57:43.439 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-07-17T11:57:43.439 INFO:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2020-07-17T11:57:43.472 INFO:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2020-07-17T11:57:43.487 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-07-17T11:57:43.547 INFO:teuthology.run_tasks:Running task kernel... 2020-07-17T11:57:43.572 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-07-17T11:57:43.573 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-07-17T11:57:43.573 INFO:teuthology.orchestra.run.smithi146:> uname -r 2020-07-17T11:57:43.579 INFO:teuthology.orchestra.run.smithi146.stdout:4.15.0-111-generic 2020-07-17T11:57:43.580 INFO:teuthology.task.kernel:Running kernel on smithi146: 4.15.0-111-generic 2020-07-17T11:57:43.580 INFO:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2020-07-17T11:57:43.675 INFO:teuthology.orchestra.run.smithi146:> sudo apt-get update 2020-07-17T11:57:43.824 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-07-17T11:57:43.949 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-17T11:57:44.029 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-07-17T11:57:44.328 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-07-17T11:57:44.472 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,011 kB] 2020-07-17T11:57:44.832 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [714 kB] 2020-07-17T11:57:44.871 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [341 kB] 2020-07-17T11:57:44.890 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,025 kB] 2020-07-17T11:57:44.965 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,092 kB] 2020-07-17T11:57:45.004 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [340 kB] 2020-07-17T11:57:45.013 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [11.5 kB] 2020-07-17T11:57:45.013 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [7,156 B] 2020-07-17T11:57:45.555 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 4,793 kB in 2s (2,878 kB/s) 2020-07-17T11:57:47.033 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2020-07-17T11:57:47.078 INFO:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-07-17T11:57:47.156 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2020-07-17T11:57:47.382 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2020-07-17T11:57:47.382 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2020-07-17T11:57:47.458 INFO:teuthology.orchestra.run.smithi146.stderr:E: Unable to locate package linux-image-current-generic 2020-07-17T11:57:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-07-17T11:57:47.460 INFO:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-07-17T11:57:47.541 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2020-07-17T11:57:47.782 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2020-07-17T11:57:47.783 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2020-07-17T11:57:47.990 INFO:teuthology.orchestra.run.smithi146.stdout:linux-image-generic is already the newest version (4.15.0.111.99). 2020-07-17T11:57:47.990 INFO:teuthology.orchestra.run.smithi146.stdout:linux-image-generic set to manually installed. 2020-07-17T11:57:47.990 INFO:teuthology.orchestra.run.smithi146.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2020-07-17T11:57:47.991 INFO:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2020-07-17T11:57:48.014 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2020-07-17T11:57:48.014 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2020-07-17T11:57:48.014 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2020-07-17T11:57:48.015 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2020-07-17T11:57:48.015 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 16 2020-07-17T11:57:48.015 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2020-07-17T11:57:48.015 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2020-07-17T11:57:48.015 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2020-07-17T11:57:48.015 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 4.15.0.111.99 2020-07-17T11:57:48.016 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.9) 2020-07-17T11:57:48.016 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-4.15.0-111-generic, linux-modules-extra-4.15.0-111-generic, linux-firmware, intel-microcode, amd64-microcode 2020-07-17T11:57:48.016 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2020-07-17T11:57:48.016 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2020-07-17T11:57:48.016 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2020-07-17T11:57:48.017 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2020-07-17T11:57:48.017 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-07-17T11:57:48.018 INFO:teuthology.orchestra.run.smithi052:> uname -r 2020-07-17T11:57:48.024 INFO:teuthology.orchestra.run.smithi052.stdout:4.15.0-111-generic 2020-07-17T11:57:48.024 INFO:teuthology.task.kernel:Running kernel on smithi052: 4.15.0-111-generic 2020-07-17T11:57:48.025 INFO:teuthology.orchestra.run.smithi052:> sudo apt-get clean 2020-07-17T11:57:48.126 INFO:teuthology.orchestra.run.smithi052:> sudo apt-get update 2020-07-17T11:57:48.269 INFO:teuthology.orchestra.run.smithi052.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-07-17T11:57:48.393 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-17T11:57:48.472 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-07-17T11:57:48.728 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-07-17T11:57:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,011 kB] 2020-07-17T11:57:49.213 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [714 kB] 2020-07-17T11:57:49.239 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [341 kB] 2020-07-17T11:57:49.249 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,092 kB] 2020-07-17T11:57:49.331 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,025 kB] 2020-07-17T11:57:49.371 INFO:teuthology.orchestra.run.smithi052.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [340 kB] 2020-07-17T11:57:49.380 INFO:teuthology.orchestra.run.smithi052.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [7,156 B] 2020-07-17T11:57:49.381 INFO:teuthology.orchestra.run.smithi052.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [11.5 kB] 2020-07-17T11:57:49.945 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 4,793 kB in 2s (2,993 kB/s) 2020-07-17T11:57:51.469 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2020-07-17T11:57:51.622 INFO:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-07-17T11:57:51.706 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2020-07-17T11:57:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2020-07-17T11:57:51.965 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2020-07-17T11:57:52.055 INFO:teuthology.orchestra.run.smithi052.stderr:E: Unable to locate package linux-image-current-generic 2020-07-17T11:57:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-07-17T11:57:52.057 INFO:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-07-17T11:57:52.136 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2020-07-17T11:57:52.385 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2020-07-17T11:57:52.386 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2020-07-17T11:57:52.600 INFO:teuthology.orchestra.run.smithi052.stdout:linux-image-generic is already the newest version (4.15.0.111.99). 2020-07-17T11:57:52.600 INFO:teuthology.orchestra.run.smithi052.stdout:linux-image-generic set to manually installed. 2020-07-17T11:57:52.601 INFO:teuthology.orchestra.run.smithi052.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2020-07-17T11:57:52.602 INFO:teuthology.orchestra.run.smithi052:> dpkg -s linux-image-generic 2020-07-17T11:57:52.628 INFO:teuthology.orchestra.run.smithi052.stdout:Package: linux-image-generic 2020-07-17T11:57:52.629 INFO:teuthology.orchestra.run.smithi052.stdout:Status: install ok installed 2020-07-17T11:57:52.629 INFO:teuthology.orchestra.run.smithi052.stdout:Priority: optional 2020-07-17T11:57:52.629 INFO:teuthology.orchestra.run.smithi052.stdout:Section: kernel 2020-07-17T11:57:52.629 INFO:teuthology.orchestra.run.smithi052.stdout:Installed-Size: 16 2020-07-17T11:57:52.630 INFO:teuthology.orchestra.run.smithi052.stdout:Maintainer: Ubuntu Kernel Team 2020-07-17T11:57:52.630 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: amd64 2020-07-17T11:57:52.630 INFO:teuthology.orchestra.run.smithi052.stdout:Source: linux-meta 2020-07-17T11:57:52.630 INFO:teuthology.orchestra.run.smithi052.stdout:Version: 4.15.0.111.99 2020-07-17T11:57:52.630 INFO:teuthology.orchestra.run.smithi052.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.9) 2020-07-17T11:57:52.631 INFO:teuthology.orchestra.run.smithi052.stdout:Depends: linux-image-4.15.0-111-generic, linux-modules-extra-4.15.0-111-generic, linux-firmware, intel-microcode, amd64-microcode 2020-07-17T11:57:52.631 INFO:teuthology.orchestra.run.smithi052.stdout:Recommends: thermald 2020-07-17T11:57:52.631 INFO:teuthology.orchestra.run.smithi052.stdout:Description: Generic Linux kernel image 2020-07-17T11:57:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: This package will always depend on the latest generic kernel image 2020-07-17T11:57:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: available. 2020-07-17T11:57:52.632 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-07-17T11:57:52.632 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-07-17T11:57:52.632 INFO:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-07-17T11:57:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2020-07-17T11:57:52.697 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-07-17T11:57:52.697 INFO:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-07-17T11:57:52.714 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2020-07-17T11:57:52.755 INFO:teuthology.run_tasks:Running task internal.base... 2020-07-17T11:57:52.766 INFO:teuthology.task.internal:Creating test directory... 2020-07-17T11:57:52.766 INFO:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-07-17T11:57:52.799 INFO:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-07-17T11:57:52.805 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-07-17T11:57:52.820 INFO:teuthology.run_tasks:Running task internal.archive... 2020-07-17T11:57:52.830 INFO:teuthology.task.internal:Creating archive directory... 2020-07-17T11:57:52.830 INFO:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-07-17T11:57:52.847 INFO:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-07-17T11:57:52.869 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-07-17T11:57:52.880 INFO:teuthology.task.internal:Enabling coredump saving... 2020-07-17T11:57:52.881 INFO:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-07-17T11:57:52.915 INFO:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-07-17T11:57:52.940 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-17T11:57:52.942 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-17T11:57:52.949 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-17T11:57:52.951 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-17T11:57:52.953 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-07-17T11:57:52.966 INFO:teuthology.task.internal:Configuring sudo... 2020-07-17T11:57:52.967 INFO:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-07-17T11:57:52.995 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-07-17T11:57:53.019 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-07-17T11:57:53.031 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-07-17T11:57:53.032 INFO:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-07-17T11:57:53.051 INFO:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-07-17T11:57:53.064 INFO:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-07-17T11:57:53.113 INFO:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-07-17T11:57:53.161 INFO:teuthology.orchestra.run.smithi146:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-07-17T11:57:53.249 INFO:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-07-17T11:57:53.256 INFO:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-07-17T11:57:53.303 INFO:teuthology.orchestra.run.smithi052:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-07-17T11:57:53.392 INFO:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2020-07-17T11:57:53.435 INFO:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2020-07-17T11:57:53.481 INFO:teuthology.run_tasks:Running task internal.timer... 2020-07-17T11:57:53.494 INFO:teuthology.task.internal:Starting timer... 2020-07-17T11:57:53.495 INFO:teuthology.run_tasks:Running task pcp... 2020-07-17T11:57:53.512 INFO:teuthology.run_tasks:Running task selinux... 2020-07-17T11:57:53.532 DEBUG:teuthology.task.selinux:Excluding smithi146: OS 'ubuntu' does not support SELinux 2020-07-17T11:57:53.532 DEBUG:teuthology.task.selinux:Excluding smithi052: OS 'ubuntu' does not support SELinux 2020-07-17T11:57:53.533 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-07-17T11:57:53.533 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-07-17T11:57:53.534 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-07-17T11:57:53.534 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-07-17T11:57:53.548 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-07-17T11:57:53.550 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-07-17T11:57:53.578 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-07-17T11:57:53.581 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi146.front.sepia.ceph.com,smithi052.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-07-17T11:57:54.056 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-07-17T11:57:55.194 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-07-17T11:57:57.788 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-07-17T11:57:57.807 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:57:57 +0000 (0:00:00.074) 0:00:00.074 *********** 2020-07-17T11:57:57.822 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:57:57 +0000 (0:00:00.015) 0:00:00.089 *********** 2020-07-17T11:57:59.612 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi146.front.sepia.ceph.com] 2020-07-17T11:57:59.662 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] 2020-07-17T11:57:59.722 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:57:59 +0000 (0:00:01.899) 0:00:01.989 *********** 2020-07-17T11:57:59.764 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:57:59 +0000 (0:00:00.041) 0:00:02.030 *********** 2020-07-17T11:58:00.117 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-07-17T11:58:00.119 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-17T11:58:00.164 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:00 +0000 (0:00:00.390) 0:00:02.421 *********** 2020-07-17T11:58:00.213 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:00 +0000 (0:00:00.059) 0:00:02.480 *********** 2020-07-17T11:58:00.808 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-07-17T11:58:00.809 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-07-17T11:58:00.948 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:00 +0000 (0:00:00.735) 0:00:03.215 *********** 2020-07-17T11:58:00.992 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-07-17T11:58:01.037 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:01 +0000 (0:00:00.088) 0:00:03.304 *********** 2020-07-17T11:58:01.313 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-07-17T11:58:01.315 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.076184", "end": "2020-07-17 11:58:01.307584", "rc": 0, "start": "2020-07-17 11:58:01.231400", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.085645", "end": "2020-07-17 11:58:00.846672", "rc": 0, "start": "2020-07-17 11:58:00.761027", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-07-17T11:58:01.351 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:01 +0000 (0:00:00.314) 0:00:03.618 *********** 2020-07-17T11:58:01.386 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:01 +0000 (0:00:00.034) 0:00:03.653 *********** 2020-07-17T11:58:01.649 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-07-17T11:58:01.651 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-17T11:58:01.691 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:01 +0000 (0:00:00.305) 0:00:03.958 *********** 2020-07-17T11:58:01.743 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:01 +0000 (0:00:00.051) 0:00:04.010 *********** 2020-07-17T11:58:02.487 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-07-17T11:58:02.489 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1594987081.7729168-173431293327678/source", "state": "file", "uid": 0} changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1594987081.7079341-117308400331801/source", "state": "file", "uid": 0} 2020-07-17T11:58:02.526 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:02 +0000 (0:00:00.782) 0:00:04.792 *********** 2020-07-17T11:58:02.567 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:02 +0000 (0:00:00.041) 0:00:04.834 *********** 2020-07-17T11:58:02.853 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-07-17T11:58:02.855 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-07-17T11:58:02.857 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-07-17T11:58:02.907 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-07-17T11:58:02.909 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-07-17T11:58:03.040 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-07-17T11:58:03.047 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-07-17T11:58:03.185 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:03 +0000 (0:00:00.617) 0:00:05.451 *********** 2020-07-17T11:58:03.220 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:03 +0000 (0:00:00.034) 0:00:05.486 *********** 2020-07-17T11:58:03.973 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi052.front.sepia.ceph.com] 2020-07-17T11:58:04.055 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-07-17T11:58:04.057 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:04 +0000 (0:00:00.835) 0:00:06.322 *********** 2020-07-17T11:58:04.479 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-07-17T11:58:04.481 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.177316", "end": "2020-07-17 11:58:04.449638", "rc": 0, "start": "2020-07-17 11:58:04.272322", "stderr": "", "stderr_lines": [], "stdout": "d2c541f972f084049b707e8806b2858a9ab4fb83", "stdout_lines": ["d2c541f972f084049b707e8806b2858a9ab4fb83"]} 2020-07-17T11:58:04.516 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:04 +0000 (0:00:00.460) 0:00:06.783 *********** 2020-07-17T11:58:04.665 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008067", "end": "2020-07-17 11:58:04.189509", "failed_when_result": false, "rc": 0, "start": "2020-07-17 11:58:04.181442", "stderr": "", "stderr_lines": [], "stdout": "d2c541f972f084049b707e8806b2858a9ab4fb83", "stdout_lines": ["d2c541f972f084049b707e8806b2858a9ab4fb83"]} 2020-07-17T11:58:04.716 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:04 +0000 (0:00:00.199) 0:00:06.983 *********** 2020-07-17T11:58:04.757 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-07-17T11:58:04.758 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-07-17T11:58:04.796 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:04 +0000 (0:00:00.079) 0:00:07.062 *********** 2020-07-17T11:58:04.833 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-07-17T11:58:04.834 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:04.870 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:04 +0000 (0:00:00.074) 0:00:07.137 *********** 2020-07-17T11:58:04.909 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-07-17T11:58:04.911 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:04.949 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:04 +0000 (0:00:00.078) 0:00:07.216 *********** 2020-07-17T11:58:04.988 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-07-17T11:58:04.990 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:04.993 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi146.front.sepia.ceph.com] 2020-07-17T11:58:05.042 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:05 +0000 (0:00:00.092) 0:00:07.309 *********** 2020-07-17T11:58:05.090 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-07-17T11:58:05.092 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.128 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:05 +0000 (0:00:00.085) 0:00:07.395 *********** 2020-07-17T11:58:05.173 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:05 +0000 (0:00:00.045) 0:00:07.440 *********** 2020-07-17T11:58:05.182 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.241 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:05 +0000 (0:00:00.067) 0:00:07.507 *********** 2020-07-17T11:58:05.344 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-07-17T11:58:05.345 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.347 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.354 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.369 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.371 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.372 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.374 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.388 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.390 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.392 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.402 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.414 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.425 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.427 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.445 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.448 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.451 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.453 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.465 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.468 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.480 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.506 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.510 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.519 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.521 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.523 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.525 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.536 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.538 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.548 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.559 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.561 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.571 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.574 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.582 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.584 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.594 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.596 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.607 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.609 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.620 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.631 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.660 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.661 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.663 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-07-17T11:58:05.670 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.199659", "end": "2020-07-17 11:58:05.610614", "rc": 0, "start": "2020-07-17 11:58:05.410955", "stderr": "", "stderr_lines": [], "stdout": "d2c541f972f084049b707e8806b2858a9ab4fb83", "stdout_lines": ["d2c541f972f084049b707e8806b2858a9ab4fb83"]} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3cker 2020-07-17T11:58:05.671 INFO:teuthology.task.ansible.out:s-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.723 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:05 +0000 (0:00:00.482) 0:00:07.989 *********** 2020-07-17T11:58:05.732 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.734 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.734 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.735 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => 2020-07-17T11:58:05.737 INFO:teuthology.task.ansible.out:(item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.738 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd71 2020-07-17T11:58:05.739 INFO:teuthology.task.ansible.out:1d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.748 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.750 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.760 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.762 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.773 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.775 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.785 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.797 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.798 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.809 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.820 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.832 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.834 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.843 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.854 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.856 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.879 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.881 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.882 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.893 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.895 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.906 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.906 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.918 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon iDR2L3LMuqFG10t1dKdXtA 61f19801e4ab7abb7597df1606036f11b0cf6a27df853dca2df8f0bb071318e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon iDR2L3LMuqFG10t1dKdXtA 61f19801e4ab7abb7597df1606036f11b0cf6a27df853dca2df8f0bb071318e1"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.925 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.942 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.953 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:05.958 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.090907", "end": "2020-07-17 11:58:05.958726", "failed_when_result": false, "rc": 0, "start": "2020-07-17 11:58:05.867819", "stderr": "", "stderr_lines": [], "stdout": "d2c541f972f084049b707e8806b2858a9ab4fb83", "stdout_lines": ["d2c541f972f084049b707e8806b2858a9ab4fb83"]} 2020-07-17T11:58:06.008 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.284) 0:00:08.274 *********** 2020-07-17T11:58:06.021 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5 2020-07-17T11:58:06.023 INFO:teuthology.task.ansible.out:c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.se 2020-07-17T11:58:06.025 INFO:teuthology.task.ansible.out:pia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.033 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.035 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.045 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-07-17T11:58:06.047 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.108 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.100) 0:00:08.374 *********** 2020-07-17T11:58:06.117 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.119 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.119 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": 2020-07-17T11:58:06.121 INFO:teuthology.task.ansible.out: "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOW 2020-07-17T11:58:06.123 INFO:teuthology.task.ansible.out:hvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.134 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.149 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-07-17T11:58:06.151 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.202 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.094) 0:00:08.469 *********** 2020-07-17T11:58:06.252 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.049) 0:00:08.518 *********** 2020-07-17T11:58:06.260 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-07-17T11:58:06.309 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.057) 0:00:08.576 *********** 2020-07-17T11:58:06.326 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.361 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.051) 0:00:08.627 *********** 2020-07-17T11:58:06.370 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.414 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.053) 0:00:08.681 *********** 2020-07-17T11:58:06.422 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-07-17T11:58:06.424 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.462 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.048) 0:00:08.729 *********** 2020-07-17T11:58:06.471 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-07-17T11:58:06.473 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.514 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.052) 0:00:08.781 *********** 2020-07-17T11:58:06.523 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.567 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.052) 0:00:08.834 *********** 2020-07-17T11:58:06.582 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.620 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.053) 0:00:08.887 *********** 2020-07-17T11:58:06.629 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.667 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.046) 0:00:08.934 *********** 2020-07-17T11:58:06.675 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-07-17T11:58:06.677 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.723 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.055) 0:00:08.990 *********** 2020-07-17T11:58:06.736 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-07-17T11:58:06.739 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.748 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.750 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.760 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.763 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.772 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.773 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.775 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.822 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:06 +0000 (0:00:00.099) 0:00:09.089 *********** 2020-07-17T11:58:06.831 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-07-17T11:58:06.833 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281a 2020-07-17T11:58:06.835 INFO:teuthology.task.ansible.out:o7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM1 2020-07-17T11:58:06.839 INFO:teuthology.task.ansible.out:3MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8RO 2020-07-17T11:58:06.846 INFO:teuthology.task.ansible.out:t6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.848 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.850 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.851 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => 2020-07-17T11:58:06.852 INFO:teuthology.task.ansible.out: (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.872 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ 2020-07-17T11:58:06.874 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", " 2020-07-17T11:58:06.876 INFO:teuthology.task.ansible.out:ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.886 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:06.887 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ 2020-07-17T11:58:06.888 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ****************************************** 2020-07-17T11:58:06.889 INFO:teuthology.task.ansible.out:** skipping: [smithi052.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.899 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:06.901 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.911 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.913 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:06.914 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.921 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.932 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.934 INFO:teuthology.task.ansible.out: 2020-07-17T11:58:06.970 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-07-17T11:58:06.973 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mkid 2020-07-17T11:58:06.975 INFO:teuthology.task.ansible.out:d', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.979 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:06.981 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.982 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.984 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:06.994 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.995 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-07-17T11:58:06.996 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.996 INFO:teuthology.task.ansible.out: 2020-07-17T11:58:06.997 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:06.998 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.018 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:07.020 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.021 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXV 2020-07-17T11:58:07.023 INFO:teuthology.task.ansible.out:SGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.023 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"an 2020-07-17T11:58:07.025 INFO:teuthology.task.ansible.out:sible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.026 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.027 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ab 2020-07-17T11:58:07.028 INFO:teuthology.task.ansible.out:hi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.037 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:07.039 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.040 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ 2020-07-17T11:58:07.041 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=liupan) => {"ansible_lo 2020-07-17T11:58:07.043 INFO:teuthology.task.ansible.out:op_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.044 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.f 2020-07-17T11:58:07.045 INFO:teuthology.task.ansible.out:ront.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.057 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.059 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.059 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.060 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.061 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item= 2020-07-17T11:58:07.064 INFO:teuthology.task.ansible.out:{'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.071 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:07.074 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.078 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-07-17T11:58:07.080 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.129 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:07 +0000 (0:00:00.306) 0:00:09.396 *********** 2020-07-17T11:58:07.142 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.146 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.146 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.147 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.149 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.149 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.150 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehl 2020-07-17T11:58:07.151 INFO:teuthology.task.ansible.out:er", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.162 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.164 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.174 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.186 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.188 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.200 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.214 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.216 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.225 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.227 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.238 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.240 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.250 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.251 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.262 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.264 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon iDR2L3LMuqFG10t1dKdXtA 61f19801e4ab7abb7597df1606036f11b0cf6a27df853dca2df8f0bb071318e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon iDR2L3LMuqFG10t1dKdXtA 61f19801e4ab7abb7597df1606036f11b0cf6a27df853dca2df8f0bb071318e1"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.274 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.276 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.286 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.297 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.309 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.311 INFO:teuthology.task.ansible.out: skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.313 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.323 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.325 INFO:teuthology.task.ansible.out: skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.334 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.348 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.352 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.353 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.355 INFO:teuthology.task.ansible.out: 2020-07-17T11:58:07.378 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.380 INFO:teuthology.task.ansible.out: 2020-07-17T11:58:07.383 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.384 INFO:teuthology.task.ansible.out: skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.386 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.388 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.398 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.401 INFO:teuthology.task.ansible.out: skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.411 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.413 INFO:teuthology.task.ansible.out: skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.434 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.436 INFO:teuthology.task.ansible.out: skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.437 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.448 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.450 INFO:teuthology.task.ansible.out: 2020-07-17T11:58:07.451 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.454 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.464 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.466 INFO:teuthology.task.ansible.out: skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.546 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:07 +0000 (0:00:00.417) 0:00:09.813 *********** 2020-07-17T11:58:07.588 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-07-17T11:58:07.627 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:07 +0000 (0:00:00.080) 0:00:09.894 *********** 2020-07-17T11:58:07.660 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-07-17T11:58:07.662 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.701 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:07 +0000 (0:00:00.074) 0:00:09.968 *********** 2020-07-17T11:58:07.730 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-07-17T11:58:07.731 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.796 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:07 +0000 (0:00:00.094) 0:00:10.063 *********** 2020-07-17T11:58:07.828 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-07-17T11:58:07.830 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "c6bfe134f92d5cc325c43b188dcccb7cdd63c6be", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-07-17T11:58:07.869 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:07 +0000 (0:00:00.072) 0:00:10.136 *********** 2020-07-17T11:58:07.877 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-07-17T11:58:07.879 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:07.918 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:07 +0000 (0:00:00.048) 0:00:10.184 *********** 2020-07-17T11:58:07.926 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi052.front.sepia.ceph.com] => { "msg": "Host smithi052.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-07-17T11:58:07.974 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:07 +0000 (0:00:00.056) 0:00:10.241 *********** 2020-07-17T11:58:07.984 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.041 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.066) 0:00:10.308 *********** 2020-07-17T11:58:08.049 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-07-17T11:58:08.051 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-07-17T11:58:08.091 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.049) 0:00:10.357 *********** 2020-07-17T11:58:08.098 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-07-17T11:58:08.100 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.142 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.051) 0:00:10.408 *********** 2020-07-17T11:58:08.149 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-07-17T11:58:08.151 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.193 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.050) 0:00:10.459 *********** 2020-07-17T11:58:08.202 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:08.203 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.204 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.206 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.218 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.219 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.220 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.222 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.265 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.072) 0:00:10.532 *********** 2020-07-17T11:58:08.272 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-17T11:58:08.274 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.286 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.287 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.288 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.299 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.300 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi146.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.343 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.078) 0:00:10.610 *********** 2020-07-17T11:58:08.370 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.372 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.373 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.384 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.395 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.397 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.399 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-07-17T11:58:08.401 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.439 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.095) 0:00:10.706 *********** 2020-07-17T11:58:08.483 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.044) 0:00:10.750 *********** 2020-07-17T11:58:08.491 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-07-17T11:58:08.493 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.534 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.050) 0:00:10.801 *********** 2020-07-17T11:58:08.577 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-07-17T11:58:08.579 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.624 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.090) 0:00:10.891 *********** 2020-07-17T11:58:08.654 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-07-17T11:58:08.656 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.697 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.073) 0:00:10.964 *********** 2020-07-17T11:58:08.737 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-07-17T11:58:08.739 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:08.788 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.090) 0:00:11.055 *********** 2020-07-17T11:58:08.827 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-07-17T11:58:08.828 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "c6bfe134f92d5cc325c43b188dcccb7cdd63c6be", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-07-17T11:58:08.869 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.081) 0:00:11.136 *********** 2020-07-17T11:58:08.877 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-07-17T11:58:08.879 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:08.932 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.063) 0:00:11.199 *********** 2020-07-17T11:58:08.942 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-07-17T11:58:08.944 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => { "msg": "Host smithi146.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-07-17T11:58:08.988 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:08 +0000 (0:00:00.055) 0:00:11.255 *********** 2020-07-17T11:58:08.995 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-07-17T11:58:08.997 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.039 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.051) 0:00:11.306 *********** 2020-07-17T11:58:09.057 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-07-17T11:58:09.059 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-07-17T11:58:09.096 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.057) 0:00:11.363 *********** 2020-07-17T11:58:09.104 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-07-17T11:58:09.106 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.159 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.061) 0:00:11.425 *********** 2020-07-17T11:58:09.167 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-07-17T11:58:09.169 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.212 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.053) 0:00:11.478 *********** 2020-07-17T11:58:09.219 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-07-17T11:58:09.221 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:09.259 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.047) 0:00:11.526 *********** 2020-07-17T11:58:09.267 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-07-17T11:58:09.268 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.305 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.045) 0:00:11.571 *********** 2020-07-17T11:58:09.311 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-07-17T11:58:09.313 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.358 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.053) 0:00:11.625 *********** 2020-07-17T11:58:09.365 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-07-17T11:58:09.381 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.412 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.054) 0:00:11.679 *********** 2020-07-17T11:58:09.421 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-07-17T11:58:09.423 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:09.461 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.048) 0:00:11.727 *********** 2020-07-17T11:58:09.469 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-07-17T11:58:09.470 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.508 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.047) 0:00:11.775 *********** 2020-07-17T11:58:09.517 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-07-17T11:58:09.520 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.567 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.059) 0:00:11.834 *********** 2020-07-17T11:58:09.576 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-07-17T11:58:09.587 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.625 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.057) 0:00:11.892 *********** 2020-07-17T11:58:09.633 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-07-17T11:58:09.635 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.677 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.052) 0:00:11.944 *********** 2020-07-17T11:58:09.686 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-07-17T11:58:09.688 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.729 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.051) 0:00:11.996 *********** 2020-07-17T11:58:09.739 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-07-17T11:58:09.740 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.792 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.063) 0:00:12.059 *********** 2020-07-17T11:58:09.809 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.846 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.053) 0:00:12.113 *********** 2020-07-17T11:58:09.857 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-07-17T11:58:09.859 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.899 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.052) 0:00:12.166 *********** 2020-07-17T11:58:09.909 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-07-17T11:58:09.916 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:09.962 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:09 +0000 (0:00:00.062) 0:00:12.228 *********** 2020-07-17T11:58:09.974 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:10.015 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.053) 0:00:12.282 *********** 2020-07-17T11:58:10.026 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-07-17T11:58:10.028 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:10.070 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.054) 0:00:12.337 *********** 2020-07-17T11:58:10.077 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-07-17T11:58:10.079 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.121 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.049) 0:00:12.387 *********** 2020-07-17T11:58:10.130 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-07-17T11:58:10.131 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.171 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.050) 0:00:12.438 *********** 2020-07-17T11:58:10.186 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.222 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.051) 0:00:12.489 *********** 2020-07-17T11:58:10.233 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-07-17T11:58:10.235 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.274 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.051) 0:00:12.540 *********** 2020-07-17T11:58:10.284 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-07-17T11:58:10.286 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.325 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.051) 0:00:12.592 *********** 2020-07-17T11:58:10.335 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-07-17T11:58:10.337 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:10.389 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.063) 0:00:12.656 *********** 2020-07-17T11:58:10.410 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.457 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.068) 0:00:12.724 *********** 2020-07-17T11:58:10.465 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-07-17T11:58:10.468 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.506 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.049) 0:00:12.773 *********** 2020-07-17T11:58:10.515 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-07-17T11:58:10.517 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.576 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.069) 0:00:12.842 *********** 2020-07-17T11:58:10.584 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-07-17T11:58:10.586 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:10.633 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.057) 0:00:12.900 *********** 2020-07-17T11:58:10.645 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.686 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.052) 0:00:12.953 *********** 2020-07-17T11:58:10.694 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-07-17T11:58:10.696 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.748 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.061) 0:00:13.014 *********** 2020-07-17T11:58:10.756 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-07-17T11:58:10.758 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:10.806 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.059) 0:00:13.073 *********** 2020-07-17T11:58:10.823 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-07-17T11:58:10.826 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.874 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.067) 0:00:13.141 *********** 2020-07-17T11:58:10.882 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-07-17T11:58:10.884 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.937 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.062) 0:00:13.204 *********** 2020-07-17T11:58:10.945 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-07-17T11:58:10.947 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:10.988 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:10 +0000 (0:00:00.051) 0:00:13.255 *********** 2020-07-17T11:58:10.995 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-07-17T11:58:10.997 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.035 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.045) 0:00:13.301 *********** 2020-07-17T11:58:11.043 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-07-17T11:58:11.045 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.080 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.045) 0:00:13.347 *********** 2020-07-17T11:58:11.089 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-07-17T11:58:11.091 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.141 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.060) 0:00:13.407 *********** 2020-07-17T11:58:11.148 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-07-17T11:58:11.149 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:11.187 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.045) 0:00:13.453 *********** 2020-07-17T11:58:11.204 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-07-17T11:58:11.207 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.248 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.061) 0:00:13.515 *********** 2020-07-17T11:58:11.257 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-07-17T11:58:11.259 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.296 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.047) 0:00:13.562 *********** 2020-07-17T11:58:11.302 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-07-17T11:58:11.304 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.345 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.049) 0:00:13.611 *********** 2020-07-17T11:58:11.352 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-07-17T11:58:11.354 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.402 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.055) 0:00:13.667 *********** 2020-07-17T11:58:11.408 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-07-17T11:58:11.412 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.463 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.062) 0:00:13.729 *********** 2020-07-17T11:58:11.471 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-07-17T11:58:11.473 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.509 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.046) 0:00:13.776 *********** 2020-07-17T11:58:11.538 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.578 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.068) 0:00:13.845 *********** 2020-07-17T11:58:11.612 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.650 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.072) 0:00:13.917 *********** 2020-07-17T11:58:11.658 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-07-17T11:58:11.666 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.012958", "end": "2020-07-17 11:58:11.152964", "rc": 0, "start": "2020-07-17 11:58:11.140006", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-07-17T11:58:11.697 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.046) 0:00:13.964 *********** 2020-07-17T11:58:11.706 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.761 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.064) 0:00:14.028 *********** 2020-07-17T11:58:11.772 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-07-17T11:58:11.774 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.830 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.068) 0:00:14.096 *********** 2020-07-17T11:58:11.846 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-07-17T11:58:11.848 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:11.886 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.056) 0:00:14.153 *********** 2020-07-17T11:58:11.895 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-07-17T11:58:11.896 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:11.937 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:11 +0000 (0:00:00.050) 0:00:14.204 *********** 2020-07-17T11:58:11.946 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-07-17T11:58:11.948 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.002 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.064) 0:00:14.269 *********** 2020-07-17T11:58:12.011 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.060 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.058) 0:00:14.327 *********** 2020-07-17T11:58:12.070 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-07-17T11:58:12.072 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.111 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.051) 0:00:14.378 *********** 2020-07-17T11:58:12.120 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-07-17T11:58:12.122 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.172 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.060) 0:00:14.439 *********** 2020-07-17T11:58:12.181 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-07-17T11:58:12.183 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.231 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.058) 0:00:14.498 *********** 2020-07-17T11:58:12.241 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-07-17T11:58:12.243 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.291 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.060) 0:00:14.558 *********** 2020-07-17T11:58:12.301 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-07-17T11:58:12.303 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.350 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.058) 0:00:14.617 *********** 2020-07-17T11:58:12.359 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-07-17T11:58:12.366 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.402 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.052) 0:00:14.669 *********** 2020-07-17T11:58:12.410 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-07-17T11:58:12.411 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.458 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.049) 0:00:14.724 *********** 2020-07-17T11:58:12.468 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-07-17T11:58:12.470 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.522 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.059) 0:00:14.784 *********** 2020-07-17T11:58:12.528 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-07-17T11:58:12.530 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.574 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.057) 0:00:14.841 *********** 2020-07-17T11:58:12.751 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.028421", "end": "2020-07-17 11:58:12.743438", "rc": 0, "start": "2020-07-17 11:58:12.715017", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-07-17T11:58:12.792 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.217) 0:00:15.059 *********** 2020-07-17T11:58:12.822 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-07-17T11:58:12.824 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.866 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.073) 0:00:15.132 *********** 2020-07-17T11:58:12.906 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-07-17T11:58:12.908 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:12.959 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:12 +0000 (0:00:00.093) 0:00:15.226 *********** 2020-07-17T11:58:12.968 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-07-17T11:58:12.970 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.319006", "end": "2020-07-17 11:58:12.500402", "rc": 0, "start": "2020-07-17 11:58:12.181396", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:58:13.008 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:13 +0000 (0:00:00.048) 0:00:15.275 *********** 2020-07-17T11:58:13.015 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-07-17T11:58:13.017 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:13.053 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:13 +0000 (0:00:00.045) 0:00:15.320 *********** 2020-07-17T11:58:13.060 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-07-17T11:58:13.062 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:13.102 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:13 +0000 (0:00:00.049) 0:00:15.369 *********** 2020-07-17T11:58:13.111 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-07-17T11:58:13.113 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:13.148 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:13 +0000 (0:00:00.046) 0:00:15.415 *********** 2020-07-17T11:58:13.179 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:13.226 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:13 +0000 (0:00:00.077) 0:00:15.493 *********** 2020-07-17T11:58:13.255 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-07-17T11:58:13.257 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:13.307 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:13 +0000 (0:00:00.081) 0:00:15.574 *********** 2020-07-17T11:58:13.315 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-07-17T11:58:13.317 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.027808", "end": "2020-07-17 11:58:12.806885", "rc": 0, "start": "2020-07-17 11:58:12.779077", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:58:13.359 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:13 +0000 (0:00:00.052) 0:00:15.626 *********** 2020-07-17T11:58:13.369 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-07-17T11:58:13.371 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:13.407 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:13 +0000 (0:00:00.047) 0:00:15.674 *********** 2020-07-17T11:58:14.507 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.952340", "end": "2020-07-17 11:58:14.500520", "rc": 0, "start": "2020-07-17 11:58:13.548180", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:58:14.545 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:14 +0000 (0:00:01.138) 0:00:16.812 *********** 2020-07-17T11:58:14.573 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-07-17T11:58:14.575 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:14.610 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:14 +0000 (0:00:00.065) 0:00:16.877 *********** 2020-07-17T11:58:14.770 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-07-17T11:58:14.771 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023619", "end": "2020-07-17 11:58:14.771985", "rc": 0, "start": "2020-07-17 11:58:14.748366", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:58:14.806 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:14 +0000 (0:00:00.196) 0:00:17.073 *********** 2020-07-17T11:58:16.684 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-07-17T11:58:16.686 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:58:16.733 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:16 +0000 (0:00:01.920) 0:00:18.994 *********** 2020-07-17T11:58:17.651 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-07-17T11:58:17.653 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:58:17.697 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:17 +0000 (0:00:00.969) 0:00:19.964 *********** 2020-07-17T11:58:17.726 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-07-17T11:58:17.728 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:17.786 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:17 +0000 (0:00:00.088) 0:00:20.052 *********** 2020-07-17T11:58:17.794 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-07-17T11:58:17.796 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:58:17.844 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:17 +0000 (0:00:00.058) 0:00:20.111 *********** 2020-07-17T11:58:18.134 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-07-17T11:58:18.136 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-07-17T11:58:18.178 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:18 +0000 (0:00:00.333) 0:00:20.444 *********** 2020-07-17T11:58:18.713 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-07-17T11:58:18.972 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} Friday 17 July 2020 11:58:18 +0000 (0:00:00.589) 0:00:21.034 *********** TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 17 July 2020 11:58:18 +0000 (0:00:00.089) 0:00:21.123 *********** 2020-07-17T11:58:19.050 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-07-17T11:58:19.052 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:58:19.088 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:19 +0000 (0:00:00.231) 0:00:21.355 *********** 2020-07-17T11:58:19.298 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-07-17T11:58:19.300 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-07-17T11:58:19.342 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:19 +0000 (0:00:00.253) 0:00:21.608 *********** 2020-07-17T11:58:19.459 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi052.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-07-17T11:58:19.796 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-07-17T11:58:20.123 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-07-17T11:58:20.184 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:20 +0000 (0:00:00.841) 0:00:22.450 *********** 2020-07-17T11:58:20.245 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-07-17T11:58:20.247 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:58:20.304 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:20 +0000 (0:00:00.120) 0:00:22.571 *********** 2020-07-17T11:58:20.322 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-07-17T11:58:20.323 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-07-17T11:58:20.377 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:20 +0000 (0:00:00.072) 0:00:22.644 *********** 2020-07-17T11:58:20.687 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-07-17T11:58:20.689 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-07-17T11:58:20.742 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-07-17T11:58:20.744 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-07-17T11:58:21.045 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-07-17T11:58:21.047 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-07-17T11:58:21.058 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-07-17T11:58:21.059 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-07-17T11:58:21.353 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-07-17T11:58:21.355 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-07-17T11:58:21.396 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:21 +0000 (0:00:01.018) 0:00:23.662 *********** 2020-07-17T11:58:21.405 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-07-17T11:58:21.407 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-07-17T11:58:21.468 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:21 +0000 (0:00:00.072) 0:00:23.734 *********** 2020-07-17T11:58:21.511 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-07-17T11:58:21.513 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-07-17T11:58:21.549 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:21 +0000 (0:00:00.081) 0:00:23.816 *********** 2020-07-17T11:58:21.557 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-07-17T11:58:21.559 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-07-17T11:58:21.615 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:21 +0000 (0:00:00.065) 0:00:23.881 *********** 2020-07-17T11:58:21.624 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-07-17T11:58:21.626 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:21.684 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:21 +0000 (0:00:00.069) 0:00:23.951 *********** 2020-07-17T11:58:21.716 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-07-17T11:58:21.719 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:21.763 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:21 +0000 (0:00:00.078) 0:00:24.030 *********** 2020-07-17T11:58:21.804 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-07-17T11:58:21.806 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:21.816 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:21.818 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:21.828 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:21.880 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:21 +0000 (0:00:00.116) 0:00:24.147 *********** 2020-07-17T11:58:21.908 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-07-17T11:58:21.910 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:21.945 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:21 +0000 (0:00:00.064) 0:00:24.212 *********** 2020-07-17T11:58:21.965 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi146.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-07-17T11:58:22.262 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-07-17T11:58:22.577 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-07-17T11:58:22.624 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:22 +0000 (0:00:00.679) 0:00:24.891 *********** 2020-07-17T11:58:22.664 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-07-17T11:58:22.666 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-07-17T11:58:22.714 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:22 +0000 (0:00:00.089) 0:00:24.981 *********** 2020-07-17T11:58:22.755 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-07-17T11:58:22.766 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:22.797 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:22 +0000 (0:00:00.082) 0:00:25.064 *********** 2020-07-17T11:58:22.827 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-07-17T11:58:22.829 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:22.886 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:22 +0000 (0:00:00.089) 0:00:25.153 *********** 2020-07-17T11:58:22.895 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-07-17T11:58:22.896 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:58:22.938 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:22 +0000 (0:00:00.051) 0:00:25.205 *********** 2020-07-17T11:58:22.949 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-07-17T11:58:22.954 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:22.994 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:22 +0000 (0:00:00.056) 0:00:25.261 *********** 2020-07-17T11:58:23.034 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-07-17T11:58:23.036 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:23.072 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:23 +0000 (0:00:00.078) 0:00:25.339 *********** 2020-07-17T11:58:23.306 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-07-17T11:58:23.352 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:23 +0000 (0:00:00.279) 0:00:25.619 *********** 2020-07-17T11:58:23.701 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-07-17T11:58:23.703 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-07-17T11:58:23.751 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:23 +0000 (0:00:00.399) 0:00:26.018 *********** 2020-07-17T11:58:23.901 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-07-17T11:58:23.903 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-07-17T11:58:23.968 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-07-17T11:58:23.970 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:58:24.018 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:24 +0000 (0:00:00.266) 0:00:26.284 *********** 2020-07-17T11:58:24.028 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-07-17T11:58:24.031 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-07-17T11:58:24.076 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:24 +0000 (0:00:00.058) 0:00:26.343 *********** 2020-07-17T11:58:24.110 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-07-17T11:58:24.112 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:24.162 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:24 +0000 (0:00:00.085) 0:00:26.428 *********** 2020-07-17T11:58:24.201 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-07-17T11:58:24.202 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:24.238 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:24 +0000 (0:00:00.076) 0:00:26.505 *********** 2020-07-17T11:58:24.240 INFO:teuthology.task.ansible.out: 2020-07-17T11:58:24.280 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-07-17T11:58:24.282 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:24.317 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:24 +0000 (0:00:00.078) 0:00:26.584 *********** 2020-07-17T11:58:24.379 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-07-17T11:58:24.381 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-07-17T11:58:24.422 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:24 +0000 (0:00:00.099) 0:00:26.683 *********** 2020-07-17T11:58:24.674 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-07-17T11:58:24.676 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-07-17T11:58:24.737 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:24 +0000 (0:00:00.315) 0:00:26.999 *********** 2020-07-17T11:58:24.788 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-07-17T11:58:24.790 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-07-17T11:58:24.864 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:24 +0000 (0:00:00.131) 0:00:27.131 *********** 2020-07-17T11:58:25.044 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-07-17T11:58:25.046 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-07-17T11:58:25.165 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-07-17T11:58:25.222 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.357) 0:00:27.488 *********** 2020-07-17T11:58:25.264 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-07-17T11:58:25.266 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:25.321 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.097) 0:00:27.586 *********** 2020-07-17T11:58:25.360 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-07-17T11:58:25.362 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-17T11:58:25.400 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.080) 0:00:27.666 *********** 2020-07-17T11:58:25.431 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-07-17T11:58:25.433 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:25.491 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.091) 0:00:27.758 *********** 2020-07-17T11:58:25.499 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-07-17T11:58:25.501 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-07-17 11:56:24 UTC", "ActiveEnterTimestampMonotonic": "17933144", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target local-fs.target system.slice basic.target -.mount systemd-tmpfiles-setup.service remote-fs.target network.target nss-lookup.target systemd-journald.socket var-run.mount sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-07-17 11:56:24 UTC", "AssertTimestampMonotonic": "17930979", "Before": "multi-user.target getty@tty1.service shutdown.target plymouth-quit.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-07-17 11:56:24 UTC", "ConditionTimestampMonotonic": "17930979", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-07-17T11:58:25.503 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1335", "ExecMainStartTimestamp": "Fri 2020-07-17 11:56:24 UTC", "ExecMainStartTimestampMonotonic": "17931969", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-07-17 11:56:24 UTC", "InactiveExitTimestampMonotonic": "17933144", "InvocationID": "8be22ab27d6e42b1b22781da18217345", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-07-17T11:58:25.505 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1335", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-07-17T11:58:25.506 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-07-17 11:56:24 UTC", "StateChangeTimestampMonotonic": "17933144", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-07-17 11:56:24 UTC", "WatchdogTimestampMonotonic": "17933143", "WatchdogUSec": "0"}} 2020-07-17T11:58:25.548 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.057) 0:00:27.815 *********** 2020-07-17T11:58:25.579 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-07-17T11:58:25.581 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:25.619 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.070) 0:00:27.886 *********** 2020-07-17T11:58:25.650 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:25.687 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.067) 0:00:27.954 *********** 2020-07-17T11:58:25.719 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-07-17T11:58:25.721 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:25.765 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.078) 0:00:28.032 *********** 2020-07-17T11:58:25.797 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:25.836 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.070) 0:00:28.103 *********** 2020-07-17T11:58:25.879 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-07-17T11:58:25.880 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:25.923 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.087) 0:00:28.190 *********** 2020-07-17T11:58:25.932 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-07-17T11:58:25.933 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-07-17T11:58:25.983 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:25 +0000 (0:00:00.060) 0:00:28.250 *********** 2020-07-17T11:58:25.992 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-07-17T11:58:25.994 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.030 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.046) 0:00:28.296 *********** 2020-07-17T11:58:26.057 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-07-17T11:58:26.059 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.110 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.080) 0:00:28.377 *********** 2020-07-17T11:58:26.151 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-07-17T11:58:26.153 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.195 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.084) 0:00:28.462 *********** 2020-07-17T11:58:26.224 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-07-17T11:58:26.225 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-07-17 11:57:04 UTC", "ActiveEnterTimestampMonotonic": "17637945", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket remote-fs.target basic.target nss-lookup.target time-sync.target sysinit.target -.mount systemd-tmpfiles-setup.service network.target var-run.mount system.slice local-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-07-17 11:57:04 UTC", "AssertTimestampMonotonic": "17636885", "Before": "xdm.service plymouth-quit.service shutdown.target getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-07-17 11:57:04 UTC", "ConditionTimestampMonotonic": "17636885", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-07-17T11:58:26.227 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1141", "ExecMainStartTimestamp": "Fri 2020-07-17 11:57:04 UTC", "ExecMainStartTimestampMonotonic": "17637920", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-07-17 11:57:04 UTC", "InactiveExitTimestampMonotonic": "17637945", "InvocationID": "490af529c70f40a8ab189ab207f5b630", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-07-17T11:58:26.230 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1141", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-07-17T11:58:26.232 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-07-17 11:57:04 UTC", "StateChangeTimestampMonotonic": "17637945", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-07-17 11:57:04 UTC", "WatchdogTimestampMonotonic": "17637944", "WatchdogUSec": "0"}} TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.264 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.068) 0:00:28.530 *********** 2020-07-17T11:58:26.302 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.038) 0:00:28.569 *********** 2020-07-17T11:58:26.311 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.360 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.058) 0:00:28.627 *********** 2020-07-17T11:58:26.368 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-07-17T11:58:26.369 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.415 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.054) 0:00:28.682 *********** 2020-07-17T11:58:26.424 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-07-17T11:58:26.426 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-07-17T11:58:26.463 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.048) 0:00:28.730 *********** 2020-07-17T11:58:26.472 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-07-17T11:58:26.474 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.513 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.049) 0:00:28.780 *********** 2020-07-17T11:58:26.520 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-07-17T11:58:26.522 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-07-17T11:58:26.574 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.053) 0:00:28.833 *********** 2020-07-17T11:58:26.575 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-07-17T11:58:26.575 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.612 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.046) 0:00:28.879 *********** 2020-07-17T11:58:26.621 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-07-17T11:58:26.658 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.045) 0:00:28.924 *********** 2020-07-17T11:58:26.666 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-07-17T11:58:26.668 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.715 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.057) 0:00:28.982 *********** 2020-07-17T11:58:26.724 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-07-17T11:58:26.729 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-07-17T11:58:26.771 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.055) 0:00:29.038 *********** 2020-07-17T11:58:26.779 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-07-17T11:58:26.781 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.830 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.058) 0:00:29.096 *********** 2020-07-17T11:58:26.870 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.907 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.078) 0:00:29.174 *********** 2020-07-17T11:58:26.915 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-07-17T11:58:26.917 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-17T11:58:26.953 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.045) 0:00:29.219 *********** 2020-07-17T11:58:26.960 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-07-17T11:58:26.962 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:26.998 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:26 +0000 (0:00:00.045) 0:00:29.265 *********** 2020-07-17T11:58:27.037 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-07-17T11:58:27.039 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:27.076 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.078) 0:00:29.343 *********** 2020-07-17T11:58:27.094 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-07-17T11:58:27.096 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "name": "ubuntu", "state": "present", "system": false} 2020-07-17T11:58:27.133 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.056) 0:00:29.400 *********** 2020-07-17T11:58:27.140 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-07-17T11:58:27.142 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:27.177 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.044) 0:00:29.444 *********** 2020-07-17T11:58:27.186 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-07-17T11:58:27.187 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:27.222 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.045) 0:00:29.489 *********** 2020-07-17T11:58:27.230 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-07-17T11:58:27.232 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:27.268 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.045) 0:00:29.535 *********** 2020-07-17T11:58:27.275 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-07-17T11:58:27.277 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:27.314 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.045) 0:00:29.581 *********** 2020-07-17T11:58:27.321 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-07-17T11:58:27.323 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-07-17T11:58:27.359 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.044) 0:00:29.626 *********** 2020-07-17T11:58:27.423 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-07-17T11:58:27.425 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-07-17T11:58:27.464 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.104) 0:00:29.731 *********** 2020-07-17T11:58:27.485 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-07-17T11:58:27.487 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-07-17T11:58:27.525 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.061) 0:00:29.792 *********** 2020-07-17T11:58:27.532 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-07-17T11:58:27.536 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-07-17T11:58:27.574 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.049) 0:00:29.841 *********** 2020-07-17T11:58:27.636 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-07-17T11:58:27.638 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-07-17T11:58:27.682 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.107) 0:00:29.949 *********** 2020-07-17T11:58:27.830 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-07-17T11:58:27.832 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-17T11:58:27.896 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:27 +0000 (0:00:00.214) 0:00:30.163 *********** 2020-07-17T11:58:28.063 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-07-17T11:58:28.065 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "name": "ubuntu", "state": "present", "system": false} 2020-07-17T11:58:28.117 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:28 +0000 (0:00:00.215) 0:00:30.378 *********** 2020-07-17T11:58:28.132 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-07-17T11:58:28.132 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-07-17T11:58:28.186 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:28 +0000 (0:00:00.074) 0:00:30.452 *********** 2020-07-17T11:58:28.198 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-07-17T11:58:28.200 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:28.243 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:28 +0000 (0:00:00.057) 0:00:30.510 *********** 2020-07-17T11:58:28.251 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-07-17T11:58:28.253 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi052"}, "changed": false} 2020-07-17T11:58:28.292 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:28 +0000 (0:00:00.049) 0:00:30.559 *********** 2020-07-17T11:58:28.300 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-07-17T11:58:28.303 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:28.340 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:28 +0000 (0:00:00.047) 0:00:30.607 *********** 2020-07-17T11:58:28.506 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-07-17T11:58:28.508 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-07-17T11:58:28.555 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:28 +0000 (0:00:00.214) 0:00:30.822 *********** 2020-07-17T11:58:29.020 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-07-17T11:58:29.022 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-07-17T11:58:29.065 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:29 +0000 (0:00:00.509) 0:00:31.332 *********** 2020-07-17T11:58:29.105 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-07-17T11:58:29.107 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi146"}, "changed": false} 2020-07-17T11:58:29.155 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:29 +0000 (0:00:00.089) 0:00:31.422 *********** 2020-07-17T11:58:29.303 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-07-17T11:58:29.305 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi052", "ansible_hostname": "smithi052", "ansible_nodename": "smithi052"}, "changed": false, "name": "smithi052"} 2020-07-17T11:58:29.340 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:29 +0000 (0:00:00.185) 0:00:31.607 *********** 2020-07-17T11:58:29.635 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi146", "ansible_hostname": "smithi146", "ansible_nodename": "smithi146"}, "changed": false, "name": "smithi146"} 2020-07-17T11:58:29.675 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:29 +0000 (0:00:00.334) 0:00:31.942 *********** 2020-07-17T11:58:29.711 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-07-17T11:58:29.746 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:29 +0000 (0:00:00.071) 0:00:32.013 *********** 2020-07-17T11:58:30.081 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-07-17T11:58:30.129 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:30 +0000 (0:00:00.382) 0:00:32.396 *********** 2020-07-17T11:58:30.159 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-07-17T11:58:30.160 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-07-17T11:58:30.198 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:30 +0000 (0:00:00.068) 0:00:32.464 *********** 2020-07-17T11:58:30.504 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-07-17T11:58:30.556 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:30 +0000 (0:00:00.358) 0:00:32.823 *********** 2020-07-17T11:58:30.731 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-07-17T11:58:30.733 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-07-17T11:58:30.795 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:30 +0000 (0:00:00.239) 0:00:33.062 *********** 2020-07-17T11:58:30.826 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-07-17T11:58:30.828 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:30.879 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:30 +0000 (0:00:00.083) 0:00:33.146 *********** 2020-07-17T11:58:30.924 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:30.958 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:30 +0000 (0:00:00.079) 0:00:33.225 *********** 2020-07-17T11:58:30.999 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-07-17T11:58:31.000 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.056 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.098) 0:00:33.323 *********** 2020-07-17T11:58:31.085 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-07-17T11:58:31.087 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.122 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.065) 0:00:33.388 *********** 2020-07-17T11:58:31.130 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-07-17T11:58:31.132 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-07-17T11:58:31.169 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.047) 0:00:33.435 *********** 2020-07-17T11:58:31.175 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-07-17T11:58:31.177 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.212 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.043) 0:00:33.479 *********** 2020-07-17T11:58:31.220 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-07-17T11:58:31.222 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.258 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.045) 0:00:33.524 *********** 2020-07-17T11:58:31.265 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-07-17T11:58:31.267 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.311 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.053) 0:00:33.578 *********** 2020-07-17T11:58:31.319 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-07-17T11:58:31.321 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.356 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.045) 0:00:33.623 *********** 2020-07-17T11:58:31.365 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.538 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.182) 0:00:33.805 *********** 2020-07-17T11:58:31.540 INFO:teuthology.task.ansible.out: 2020-07-17T11:58:31.560 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-07-17T11:58:31.562 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.600 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.061) 0:00:33.867 *********** 2020-07-17T11:58:31.608 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-07-17T11:58:31.611 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.659 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.059) 0:00:33.926 *********** 2020-07-17T11:58:31.673 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-07-17T11:58:31.675 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.726 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.067) 0:00:33.993 *********** 2020-07-17T11:58:31.736 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-07-17T11:58:31.780 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.053) 0:00:34.047 *********** 2020-07-17T11:58:31.792 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.828 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.047) 0:00:34.095 *********** 2020-07-17T11:58:31.837 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-07-17T11:58:31.883 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.055) 0:00:34.150 *********** 2020-07-17T11:58:31.894 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-07-17T11:58:31.896 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:31.932 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.048) 0:00:34.198 *********** 2020-07-17T11:58:31.941 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-07-17T11:58:31.984 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:31 +0000 (0:00:00.052) 0:00:34.251 *********** 2020-07-17T11:58:31.994 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-07-17T11:58:31.996 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.049 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.064) 0:00:34.316 *********** 2020-07-17T11:58:32.059 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-07-17T11:58:32.061 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.104 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.054) 0:00:34.370 *********** 2020-07-17T11:58:32.112 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-07-17T11:58:32.114 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.170 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.066) 0:00:34.437 *********** 2020-07-17T11:58:32.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-07-17T11:58:32.179 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.222 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.051) 0:00:34.488 *********** 2020-07-17T11:58:32.232 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-07-17T11:58:32.282 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.055) 0:00:34.544 *********** 2020-07-17T11:58:32.295 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.336 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.058) 0:00:34.603 *********** 2020-07-17T11:58:32.344 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-07-17T11:58:32.397 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.061) 0:00:34.664 *********** 2020-07-17T11:58:32.406 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:58:32.408 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.446 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.048) 0:00:34.712 *********** 2020-07-17T11:58:32.453 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-07-17T11:58:32.490 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.044) 0:00:34.757 *********** 2020-07-17T11:58:32.499 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-07-17T11:58:32.501 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.539 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.049) 0:00:34.806 *********** 2020-07-17T11:58:32.551 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.594 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.050) 0:00:34.857 *********** 2020-07-17T11:58:32.601 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-07-17T11:58:32.603 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.642 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.051) 0:00:34.909 *********** 2020-07-17T11:58:32.652 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-07-17T11:58:32.653 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.710 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.061) 0:00:34.971 *********** 2020-07-17T11:58:32.712 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.760 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.056) 0:00:35.027 *********** 2020-07-17T11:58:32.768 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-07-17T11:58:32.770 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.815 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.054) 0:00:35.082 *********** 2020-07-17T11:58:32.824 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-07-17T11:58:32.826 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.874 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.059) 0:00:35.141 *********** 2020-07-17T11:58:32.885 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.930 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.055) 0:00:35.197 *********** 2020-07-17T11:58:32.942 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-07-17T11:58:32.944 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:32.992 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:32 +0000 (0:00:00.060) 0:00:35.258 *********** 2020-07-17T11:58:33.000 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-07-17T11:58:33.002 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.044 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.053) 0:00:35.311 *********** 2020-07-17T11:58:33.051 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-17T11:58:33.053 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.091 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.046) 0:00:35.358 *********** 2020-07-17T11:58:33.108 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.146 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.055) 0:00:35.413 *********** 2020-07-17T11:58:33.155 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-17T11:58:33.155 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.215 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.069) 0:00:35.482 *********** 2020-07-17T11:58:33.226 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-07-17T11:58:33.228 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.265 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.049) 0:00:35.532 *********** 2020-07-17T11:58:33.274 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-07-17T11:58:33.276 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.336 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.071) 0:00:35.603 *********** 2020-07-17T11:58:33.345 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-07-17T11:58:33.348 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.391 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.054) 0:00:35.658 *********** 2020-07-17T11:58:33.400 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-07-17T11:58:33.402 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.451 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.060) 0:00:35.718 *********** 2020-07-17T11:58:33.462 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-07-17T11:58:33.464 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.509 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.052) 0:00:35.770 *********** 2020-07-17T11:58:33.518 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-17T11:58:33.520 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.560 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.056) 0:00:35.827 *********** 2020-07-17T11:58:33.570 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-17T11:58:33.572 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.624 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.063) 0:00:35.891 *********** 2020-07-17T11:58:33.633 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-07-17T11:58:33.635 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.677 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.053) 0:00:35.944 *********** 2020-07-17T11:58:33.687 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-17T11:58:33.687 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.741 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.063) 0:00:36.007 *********** 2020-07-17T11:58:33.751 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-07-17T11:58:33.753 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.797 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.056) 0:00:36.064 *********** 2020-07-17T11:58:33.810 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-07-17T11:58:33.811 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.864 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.066) 0:00:36.131 *********** 2020-07-17T11:58:33.872 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-07-17T11:58:33.874 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.921 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.056) 0:00:36.188 *********** 2020-07-17T11:58:33.934 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-07-17T11:58:33.936 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:33.978 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:33 +0000 (0:00:00.057) 0:00:36.245 *********** 2020-07-17T11:58:33.988 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-07-17T11:58:33.990 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.040 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.061) 0:00:36.306 *********** 2020-07-17T11:58:34.048 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-17T11:58:34.050 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.094 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.054) 0:00:36.361 *********** 2020-07-17T11:58:34.106 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-07-17T11:58:34.108 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.166 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.066) 0:00:36.428 *********** 2020-07-17T11:58:34.171 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.211 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.050) 0:00:36.478 *********** 2020-07-17T11:58:34.221 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-07-17T11:58:34.225 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.271 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.059) 0:00:36.538 *********** 2020-07-17T11:58:34.285 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-07-17T11:58:34.287 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.338 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.066) 0:00:36.605 *********** 2020-07-17T11:58:34.349 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-07-17T11:58:34.351 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.411 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.072) 0:00:36.678 *********** 2020-07-17T11:58:34.421 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-07-17T11:58:34.423 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.465 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.054) 0:00:36.732 *********** 2020-07-17T11:58:34.474 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-07-17T11:58:34.476 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.531 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.066) 0:00:36.798 *********** 2020-07-17T11:58:34.544 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-07-17T11:58:34.545 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.598 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.066) 0:00:36.865 *********** 2020-07-17T11:58:34.640 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.694 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.096) 0:00:36.961 *********** 2020-07-17T11:58:34.696 INFO:teuthology.task.ansible.out: 2020-07-17T11:58:34.706 INFO:teuthology.task.ansible.out: 2020-07-17T11:58:34.708 INFO:teuthology.task.ansible.out:TASK [testnode : Check for custom repos] *************************************** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005291", "end": "2020-07-17 11:58:34.244622", "rc": 0, "start": "2020-07-17 11:58:34.239331", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:58:34.761 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.066) 0:00:37.028 *********** 2020-07-17T11:58:34.771 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-07-17T11:58:34.773 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.811 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.050) 0:00:37.078 *********** 2020-07-17T11:58:34.820 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-07-17T11:58:34.863 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.051) 0:00:37.130 *********** 2020-07-17T11:58:34.871 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-07-17T11:58:34.873 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.909 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.045) 0:00:37.175 *********** 2020-07-17T11:58:34.964 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:34.998 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:34 +0000 (0:00:00.089) 0:00:37.265 *********** 2020-07-17T11:58:35.157 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-07-17T11:58:35.159 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005145", "end": "2020-07-17 11:58:35.157863", "rc": 0, "start": "2020-07-17 11:58:35.152718", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:58:35.193 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:35 +0000 (0:00:00.195) 0:00:37.460 *********** 2020-07-17T11:58:35.221 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-07-17T11:58:35.223 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} TASK [testnode : Remove custom repos] ****************************************** 2020-07-17T11:58:35.263 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:35 +0000 (0:00:00.070) 0:00:37.530 *********** 2020-07-17T11:58:35.308 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:35 +0000 (0:00:00.044) 0:00:37.575 *********** 2020-07-17T11:58:35.328 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-07-17T11:58:35.330 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:35.377 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:35 +0000 (0:00:00.068) 0:00:37.643 *********** 2020-07-17T11:58:35.687 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-07-17T11:58:35.689 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-07-17T11:58:35.729 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:35 +0000 (0:00:00.352) 0:00:37.996 *********** 2020-07-17T11:58:35.771 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-07-17T11:58:35.773 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:35.811 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:35 +0000 (0:00:00.081) 0:00:38.078 *********** 2020-07-17T11:58:36.702 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-07-17T11:58:36.704 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-07-17T11:58:36.830 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-07-17T11:58:37.663 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-07-17T11:58:37.706 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:37 +0000 (0:00:01.894) 0:00:39.973 *********** 2020-07-17T11:58:37.804 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-07-17T11:58:37.852 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:37 +0000 (0:00:00.145) 0:00:40.119 *********** 2020-07-17T11:58:38.634 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:58:38.669 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:38 +0000 (0:00:00.817) 0:00:40.936 *********** 2020-07-17T11:58:38.704 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-07-17T11:58:38.743 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:38 +0000 (0:00:00.072) 0:00:41.009 *********** 2020-07-17T11:58:38.764 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-07-17T11:58:38.766 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:58:38.806 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:38 +0000 (0:00:00.063) 0:00:41.073 *********** 2020-07-17T11:58:38.879 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-07-17T11:58:38.949 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:38 +0000 (0:00:00.142) 0:00:41.215 *********** 2020-07-17T11:58:38.957 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-07-17T11:58:38.960 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.037385", "end": "2020-07-17 11:58:38.463372", "rc": 0, "start": "2020-07-17 11:58:38.425987", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:58:39.032 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:39 +0000 (0:00:00.083) 0:00:41.299 *********** 2020-07-17T11:58:39.147 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034373", "end": "2020-07-17 11:58:39.135193", "rc": 0, "start": "2020-07-17 11:58:39.100820", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:58:39.208 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:39 +0000 (0:00:00.175) 0:00:41.474 *********** 2020-07-17T11:58:42.028 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-07-17T11:58:42.030 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:58:42.086 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:42 +0000 (0:00:02.877) 0:00:44.352 *********** 2020-07-17T11:58:42.149 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:58:42.217 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:42 +0000 (0:00:00.131) 0:00:44.484 *********** 2020-07-17T11:58:51.269 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-07-17T11:58:51.271 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 1 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102052 files and directories 2020-07-17T11:58:51.273 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 1 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-07-17T11:58:51.275 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102052 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-17T11:58:51.323 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:51 +0000 (0:00:09.105) 0:00:53.590 *********** 2020-07-17T11:58:51.831 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 1 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102052 files and directories 2020-07-17T11:58:51.833 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 1 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-07-17T11:58:51.835 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102052 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-17T11:58:51.869 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:51 +0000 (0:00:00.545) 0:00:54.135 *********** 2020-07-17T11:58:52.231 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-17T11:58:52.233 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:58:52.273 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:52 +0000 (0:00:00.403) 0:00:54.539 *********** 2020-07-17T11:58:52.313 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-07-17T11:58:52.315 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:52.354 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:52 +0000 (0:00:00.080) 0:00:54.620 *********** 2020-07-17T11:58:52.796 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-17T11:58:52.798 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:58:52.854 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:52 +0000 (0:00:00.500) 0:00:55.121 *********** 2020-07-17T11:58:52.895 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:58:52.958 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:58:52 +0000 (0:00:00.103) 0:00:55.225 *********** 2020-07-17T11:58:52.960 INFO:teuthology.task.ansible.out: 2020-07-17T11:59:00.932 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-17T11:59:00.934 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]\nFetched 10.4 MB in 1s (9558 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-07-17T11:59:00.935 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101910 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-07-17T11:59:00.937 INFO:teuthology.task.ansible.out:1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]", "Fetched 10.4 MB in 1s (9558 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-07-17T11:59:00.939 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101910 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-17T11:59:00.975 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:00 +0000 (0:00:08.016) 0:01:03.241 *********** 2020-07-17T11:59:01.863 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:59:01.931 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:01 +0000 (0:00:00.955) 0:01:04.197 *********** 2020-07-17T11:59:02.080 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-07-17T11:59:02.082 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-07-17T11:59:02.130 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:02 +0000 (0:00:00.198) 0:01:04.396 *********** 2020-07-17T11:59:02.174 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:02.228 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:02 +0000 (0:00:00.098) 0:01:04.495 *********** 2020-07-17T11:59:02.249 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-17T11:59:02.250 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]\nFetched 10.4 MB in 1s (9768 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-07-17T11:59:02.252 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101910 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-07-17T11:59:02.261 INFO:teuthology.task.ansible.out:1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]", "Fetched 10.4 MB in 1s (9768 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-07-17T11:59:02.263 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101910 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-17T11:59:02.294 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:02 +0000 (0:00:00.065) 0:01:04.561 *********** 2020-07-17T11:59:02.411 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-07-17T11:59:02.413 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-07-17T11:59:02.448 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:02 +0000 (0:00:00.154) 0:01:04.715 *********** 2020-07-17T11:59:02.817 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-07-17T11:59:02.862 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:02 +0000 (0:00:00.413) 0:01:05.129 *********** 2020-07-17T11:59:03.045 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-07-17T11:59:03.047 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1224, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-07-17T11:59:03.112 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.250) 0:01:05.379 *********** 2020-07-17T11:59:03.205 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-07-17T11:59:03.207 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:59:03.253 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.140) 0:01:05.520 *********** 2020-07-17T11:59:03.286 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-07-17T11:59:03.288 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-17T11:59:03.339 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.086) 0:01:05.606 *********** 2020-07-17T11:59:03.401 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-07-17T11:59:03.443 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.103) 0:01:05.710 *********** 2020-07-17T11:59:03.474 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-07-17T11:59:03.478 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:03.530 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.087) 0:01:05.797 *********** 2020-07-17T11:59:03.680 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-07-17T11:59:03.681 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2020-07-17 11:57:04 UTC", "ActiveEnterTimestampMonotonic": "18052547", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket systemd-tmpfiles-setup.service system.slice nss-lookup.target -.mount sysinit.target network.target basic.target remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-07-17 11:57:04 UTC", "AssertTimestampMonotonic": "17636255", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-07-17 11:57:04 UTC", "ConditionTimestampMonotonic": "17636255", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-07-17T11:59:03.683 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1223", "ExecMainStartTimestamp": "Fri 2020-07-17 11:57:04 UTC", "ExecMainStartTimestampMonotonic": "18052536", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-07-17 11:57:04 UTC", "InactiveExitTimestampMonotonic": "17636843", "InvocationID": "154ceace3681423181c7ffb6ffef5b47", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-07-17T11:59:03.685 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1223", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-07-17T11:59:03.687 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-07-17 11:57:04 UTC", "StateChangeTimestampMonotonic": "18052547", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-07-17 11:57:04 UTC", "WatchdogTimestampMonotonic": "18052546", "WatchdogUSec": "0"}} TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-07-17T11:59:03.719 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.188) 0:01:05.986 *********** 2020-07-17T11:59:03.761 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.041) 0:01:06.027 *********** 2020-07-17T11:59:03.796 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:03.851 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.090) 0:01:06.118 *********** 2020-07-17T11:59:03.881 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-07-17T11:59:03.883 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:03.921 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.069) 0:01:06.188 *********** 2020-07-17T11:59:03.950 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-07-17T11:59:03.952 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:03.997 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:03 +0000 (0:00:00.075) 0:01:06.264 *********** 2020-07-17T11:59:04.025 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.067 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.070) 0:01:06.334 *********** 2020-07-17T11:59:04.087 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-07-17T11:59:04.125 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.058) 0:01:06.392 *********** 2020-07-17T11:59:04.136 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.182 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.056) 0:01:06.449 *********** 2020-07-17T11:59:04.213 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-07-17T11:59:04.216 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.274 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.091) 0:01:06.541 *********** 2020-07-17T11:59:04.304 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-07-17T11:59:04.306 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1224, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-07-17T11:59:04.310 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.357 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.083) 0:01:06.624 *********** 2020-07-17T11:59:04.412 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.054) 0:01:06.679 *********** 2020-07-17T11:59:04.441 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-17T11:59:04.443 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.479 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.067) 0:01:06.746 *********** 2020-07-17T11:59:04.509 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:59:04.511 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-17T11:59:04.551 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.071) 0:01:06.818 *********** 2020-07-17T11:59:04.587 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.036) 0:01:06.854 *********** 2020-07-17T11:59:04.618 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.670 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.082) 0:01:06.937 *********** 2020-07-17T11:59:04.700 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-17T11:59:04.702 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.737 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.066) 0:01:07.003 *********** 2020-07-17T11:59:04.764 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-17T11:59:04.766 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.801 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.064) 0:01:07.068 *********** 2020-07-17T11:59:04.841 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-07-17T11:59:04.842 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.898 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.097) 0:01:07.165 *********** 2020-07-17T11:59:04.909 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-07-17T11:59:04.911 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2020-07-17 11:56:25 UTC", "ActiveEnterTimestampMonotonic": "18329177", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice sysinit.target remote-fs.target network.target systemd-journald.socket systemd-tmpfiles-setup.service -.mount basic.target nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-07-17 11:56:24 UTC", "AssertTimestampMonotonic": "17928882", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-07-17 11:56:24 UTC", "ConditionTimestampMonotonic": "17928882", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-07-17T11:59:04.912 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1414", "ExecMainStartTimestamp": "Fri 2020-07-17 11:56:25 UTC", "ExecMainStartTimestampMonotonic": "18329165", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-07-17 11:56:24 UTC", "InactiveExitTimestampMonotonic": "17930034", "InvocationID": "44defc82b7334cec8d2c70dcae0808c8", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-07-17T11:59:04.914 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1414", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-07-17T11:59:04.916 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-07-17 11:56:25 UTC", "StateChangeTimestampMonotonic": "18329177", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-07-17 11:56:25 UTC", "WatchdogTimestampMonotonic": "18329175", "WatchdogUSec": "0"}} 2020-07-17T11:59:04.950 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.051) 0:01:07.217 *********** 2020-07-17T11:59:04.958 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-17T11:59:04.960 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:04.999 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:04 +0000 (0:00:00.048) 0:01:07.265 *********** 2020-07-17T11:59:05.006 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-07-17T11:59:05.007 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.067 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.068) 0:01:07.334 *********** 2020-07-17T11:59:05.078 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-17T11:59:05.080 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.119 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.052) 0:01:07.386 *********** 2020-07-17T11:59:05.127 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-07-17T11:59:05.129 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.172 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.053) 0:01:07.439 *********** 2020-07-17T11:59:05.181 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-17T11:59:05.183 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.221 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.048) 0:01:07.488 *********** 2020-07-17T11:59:05.237 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-07-17T11:59:05.239 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.273 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.052) 0:01:07.540 *********** 2020-07-17T11:59:05.281 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-07-17T11:59:05.283 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.319 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.046) 0:01:07.586 *********** 2020-07-17T11:59:05.327 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-07-17T11:59:05.328 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.366 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.045) 0:01:07.631 *********** 2020-07-17T11:59:05.373 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-07-17T11:59:05.374 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.413 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.048) 0:01:07.679 *********** 2020-07-17T11:59:05.420 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-07-17T11:59:05.422 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.458 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.045) 0:01:07.725 *********** 2020-07-17T11:59:05.466 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-07-17T11:59:05.467 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.513 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.053) 0:01:07.779 *********** 2020-07-17T11:59:05.522 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.561 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.048) 0:01:07.827 *********** 2020-07-17T11:59:05.568 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-17T11:59:05.571 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.607 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.045) 0:01:07.873 *********** 2020-07-17T11:59:05.614 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-07-17T11:59:05.616 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.660 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.053) 0:01:07.927 *********** 2020-07-17T11:59:05.667 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-17T11:59:05.670 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.707 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.047) 0:01:07.974 *********** 2020-07-17T11:59:05.715 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-17T11:59:05.717 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.763 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.055) 0:01:08.030 *********** 2020-07-17T11:59:05.773 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.811 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.047) 0:01:08.078 *********** 2020-07-17T11:59:05.820 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:59:05.822 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.863 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.052) 0:01:08.130 *********** 2020-07-17T11:59:05.871 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-17T11:59:05.873 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.921 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.058) 0:01:08.188 *********** 2020-07-17T11:59:05.931 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-07-17T11:59:05.933 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:05.973 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:05 +0000 (0:00:00.051) 0:01:08.240 *********** 2020-07-17T11:59:05.985 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-17T11:59:05.987 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.030 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.056) 0:01:08.296 *********** 2020-07-17T11:59:06.051 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-17T11:59:06.052 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.104 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.074) 0:01:08.371 *********** 2020-07-17T11:59:06.114 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.155 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.051) 0:01:08.422 *********** 2020-07-17T11:59:06.164 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-17T11:59:06.166 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.204 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.049) 0:01:08.471 *********** 2020-07-17T11:59:06.214 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-07-17T11:59:06.215 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.259 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.054) 0:01:08.526 *********** 2020-07-17T11:59:06.268 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-07-17T11:59:06.270 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.306 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.046) 0:01:08.573 *********** 2020-07-17T11:59:06.345 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-17T11:59:06.347 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.381 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.074) 0:01:08.648 *********** 2020-07-17T11:59:06.400 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-07-17T11:59:06.409 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-07-17T11:59:06.435 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.054) 0:01:08.702 *********** 2020-07-17T11:59:06.442 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-17T11:59:06.444 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.484 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.045) 0:01:08.748 *********** 2020-07-17T11:59:06.510 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-17T11:59:06.512 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.546 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.065) 0:01:08.813 *********** 2020-07-17T11:59:06.576 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.613 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.066) 0:01:08.880 *********** 2020-07-17T11:59:06.642 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-07-17T11:59:06.662 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.685 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.072) 0:01:08.952 *********** 2020-07-17T11:59:06.715 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-07-17T11:59:06.717 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.767 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.081) 0:01:09.033 *********** 2020-07-17T11:59:06.797 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-17T11:59:06.799 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.839 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.072) 0:01:09.106 *********** 2020-07-17T11:59:06.849 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-07-17T11:59:06.851 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-07-17T11:59:06.896 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.056) 0:01:09.163 *********** 2020-07-17T11:59:06.910 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:06.955 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:06 +0000 (0:00:00.058) 0:01:09.221 *********** 2020-07-17T11:59:06.966 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-07-17T11:59:06.968 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:07.009 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:07 +0000 (0:00:00.053) 0:01:09.275 *********** 2020-07-17T11:59:07.018 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-17T11:59:07.020 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:07.069 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:07 +0000 (0:00:00.059) 0:01:09.335 *********** 2020-07-17T11:59:07.114 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-17T11:59:07.116 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:07.163 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:07 +0000 (0:00:00.094) 0:01:09.430 *********** 2020-07-17T11:59:07.216 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-17T11:59:07.218 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:07.267 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:07 +0000 (0:00:00.104) 0:01:09.534 *********** 2020-07-17T11:59:07.308 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-17T11:59:07.314 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:07.352 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:07 +0000 (0:00:00.084) 0:01:09.619 *********** 2020-07-17T11:59:07.385 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-07-17T11:59:07.394 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:07.438 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:07 +0000 (0:00:00.085) 0:01:09.704 *********** 2020-07-17T11:59:07.512 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-07-17T11:59:07.514 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-07-17T11:59:07.561 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:07 +0000 (0:00:00.123) 0:01:09.827 *********** 2020-07-17T11:59:07.603 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-07-17T11:59:07.605 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-07-17T11:59:07.655 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:07 +0000 (0:00:00.094) 0:01:09.922 *********** 2020-07-17T11:59:07.950 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-07-17T11:59:07.995 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:07 +0000 (0:00:00.340) 0:01:10.262 *********** 2020-07-17T11:59:08.043 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-07-17T11:59:08.045 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-07-17T11:59:08.102 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:08 +0000 (0:00:00.106) 0:01:10.369 *********** 2020-07-17T11:59:08.132 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:08.183 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:08 +0000 (0:00:00.081) 0:01:10.450 *********** 2020-07-17T11:59:08.393 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-07-17T11:59:08.436 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:08 +0000 (0:00:00.252) 0:01:10.703 *********** 2020-07-17T11:59:08.593 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-07-17T11:59:08.594 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-17T11:59:08.631 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:08 +0000 (0:00:00.194) 0:01:10.897 *********** 2020-07-17T11:59:08.639 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-07-17T11:59:08.641 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-07-17T11:59:08.680 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:08 +0000 (0:00:00.049) 0:01:10.947 *********** 2020-07-17T11:59:08.687 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-07-17T11:59:08.689 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:08.749 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:08 +0000 (0:00:00.068) 0:01:11.016 *********** 2020-07-17T11:59:08.797 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-07-17T11:59:08.802 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:08.869 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:08 +0000 (0:00:00.120) 0:01:11.136 *********** 2020-07-17T11:59:08.934 INFO:teuthology.task.ansible.out: 2020-07-17T11:59:08.936 INFO:teuthology.task.ansible.out:TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:08.994 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:08 +0000 (0:00:00.124) 0:01:11.260 *********** 2020-07-17T11:59:09.048 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-07-17T11:59:09.050 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.088 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.094) 0:01:11.355 *********** 2020-07-17T11:59:09.096 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-07-17T11:59:09.098 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-07-17T11:59:09.147 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.058) 0:01:11.414 *********** 2020-07-17T11:59:09.156 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-07-17T11:59:09.158 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.195 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.048) 0:01:11.462 *********** 2020-07-17T11:59:09.242 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:59:09.244 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-07-17T11:59:09.286 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.091) 0:01:11.553 *********** 2020-07-17T11:59:09.352 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-07-17T11:59:09.353 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.363 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.374 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.385 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.387 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.414 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi146.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.420 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.422 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.434 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.446 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.463 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.504 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.218) 0:01:11.771 *********** 2020-07-17T11:59:09.526 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-07-17T11:59:09.528 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-07-17T11:59:09.571 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.065) 0:01:11.837 *********** 2020-07-17T11:59:09.741 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-07-17T11:59:09.743 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-17T11:59:09.796 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.221) 0:01:12.059 *********** 2020-07-17T11:59:09.802 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-07-17T11:59:09.804 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.007418", "end": "2020-07-17 11:59:09.759140", "rc": 0, "start": "2020-07-17 11:59:09.751722", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-07-17T11:59:09.850 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.057) 0:01:12.117 *********** 2020-07-17T11:59:09.858 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-07-17T11:59:09.861 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.906 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.055) 0:01:12.172 *********** 2020-07-17T11:59:09.917 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-07-17T11:59:09.918 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:09.965 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:09 +0000 (0:00:00.059) 0:01:12.232 *********** 2020-07-17T11:59:09.975 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.014 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:10 +0000 (0:00:00.049) 0:01:12.281 *********** 2020-07-17T11:59:10.022 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-07-17T11:59:10.023 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.064 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:10 +0000 (0:00:00.049) 0:01:12.331 *********** 2020-07-17T11:59:10.074 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-07-17T11:59:10.075 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.120 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:10 +0000 (0:00:00.055) 0:01:12.387 *********** 2020-07-17T11:59:10.135 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.172 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:10 +0000 (0:00:00.051) 0:01:12.439 *********** 2020-07-17T11:59:10.181 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-07-17T11:59:10.183 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.221 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:10 +0000 (0:00:00.048) 0:01:12.487 *********** 2020-07-17T11:59:10.282 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-07-17T11:59:10.285 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.324 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:10 +0000 (0:00:00.103) 0:01:12.591 *********** 2020-07-17T11:59:10.375 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:59:10.377 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-07-17T11:59:10.419 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:10 +0000 (0:00:00.095) 0:01:12.686 *********** 2020-07-17T11:59:10.494 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.498 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.521 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.532 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.544 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-07-17T11:59:10.558 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.560 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.571 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.582 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.593 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.606 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:10.649 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:10 +0000 (0:00:00.230) 0:01:12.916 *********** 2020-07-17T11:59:10.903 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-07-17T11:59:10.904 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.005452", "end": "2020-07-17 11:59:10.874185", "rc": 0, "start": "2020-07-17 11:59:10.868733", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-07-17T11:59:10.952 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:10 +0000 (0:00:00.302) 0:01:13.219 *********** 2020-07-17T11:59:11.004 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:11.052 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:11 +0000 (0:00:00.100) 0:01:13.319 *********** 2020-07-17T11:59:11.074 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-07-17T11:59:11.076 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:59:11.110 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:11 +0000 (0:00:00.057) 0:01:13.377 *********** 2020-07-17T11:59:11.118 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-07-17T11:59:11.120 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:11.170 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:11 +0000 (0:00:00.055) 0:01:13.433 *********** 2020-07-17T11:59:11.174 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-07-17T11:59:11.175 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:11.212 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:11 +0000 (0:00:00.046) 0:01:13.479 *********** 2020-07-17T11:59:11.220 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-07-17T11:59:11.222 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:11.259 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:11 +0000 (0:00:00.046) 0:01:13.526 *********** 2020-07-17T11:59:11.271 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-07-17T11:59:11.278 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'block_used': 13636014, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226459094, 'size_available': 927576449024, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113847, 'block_size': 4096, 'inode_available': 60941129}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226459094, "block_size": 4096, "block_total": 240095108, "block_used": 13636014, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941129, "inode_total": 61054976, "inode_used": 113847, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927576449024, "size_total": 983429562368, "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}} 2020-07-17T11:59:11.316 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:11 +0000 (0:00:00.056) 0:01:13.583 *********** 2020-07-17T11:59:11.486 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009792", "end": "2020-07-17 11:59:11.486243", "rc": 0, "start": "2020-07-17 11:59:11.476451", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:11.553 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:11 +0000 (0:00:00.235) 0:01:13.818 *********** 2020-07-17T11:59:11.610 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-07-17T11:59:11.612 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'block_used': 13636014, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226459094, 'size_available': 927576449024, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113847, 'block_size': 4096, 'inode_available': 60941129}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226459094, "block_size": 4096, "block_total": 240095108, "block_used": 13636014, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941129, "inode_total": 61054976, "inode_used": 113847, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927576449024, "size_total": 983429562368, "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:11.657 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:11 +0000 (0:00:00.105) 0:01:13.924 *********** 2020-07-17T11:59:11.700 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-07-17T11:59:11.737 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:11 +0000 (0:00:00.080) 0:01:14.004 *********** 2020-07-17T11:59:11.801 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-07-17T11:59:11.803 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:12.186 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:59:12.225 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:12 +0000 (0:00:00.487) 0:01:14.491 *********** 2020-07-17T11:59:12.268 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:12.307 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:12 +0000 (0:00:00.082) 0:01:14.574 *********** 2020-07-17T11:59:12.382 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-07-17T11:59:12.384 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'block_used': 13636026, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226459082, 'size_available': 927576399872, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113846, 'block_size': 4096, 'inode_available': 60941130}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226459082, "block_size": 4096, "block_total": 240095108, "block_used": 13636026, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941130, "inode_total": 61054976, "inode_used": 113846, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927576399872, "size_total": 983429562368, "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}} 2020-07-17T11:59:12.464 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:12 +0000 (0:00:00.157) 0:01:14.731 *********** 2020-07-17T11:59:12.617 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-07-17T11:59:12.624 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.011240", "end": "2020-07-17 11:59:12.178585", "rc": 0, "start": "2020-07-17 11:59:12.167345", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:12.656 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:12 +0000 (0:00:00.191) 0:01:14.923 *********** 2020-07-17T11:59:12.697 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-07-17T11:59:12.699 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'block_used': 13636026, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226459082, 'size_available': 927576399872, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113846, 'block_size': 4096, 'inode_available': 60941130}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226459082, "block_size": 4096, "block_total": 240095108, "block_used": 13636026, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941130, "inode_total": 61054976, "inode_used": 113846, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927576399872, "size_total": 983429562368, "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:12.738 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:12 +0000 (0:00:00.081) 0:01:15.005 *********** 2020-07-17T11:59:12.786 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-07-17T11:59:12.826 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:12 +0000 (0:00:00.088) 0:01:15.093 *********** 2020-07-17T11:59:12.890 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-07-17T11:59:12.892 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:13.590 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2BfuFHOiZ83arGugHODhQIVsj3AATeCpFuF'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.665795", "end": "2020-07-17 11:59:13.588243", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2BfuFHOiZ83arGugHODhQIVsj3AATeCpFuF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:11.922448", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:13.591 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:13.634 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:14.855 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bf0mYq5l9ctiJdxZoUVyJnonj2wTuVgg08'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.118693", "end": "2020-07-17 11:59:14.854059", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bf0mYq5l9ctiJdxZoUVyJnonj2wTuVgg08"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:13.735366", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:14.867 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFPH', 'wwn-0x5000c50092925063'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFPH-part1', 'wwn-0x5000c50092925063-part1'], 'uuids': ['986a1d3c-fee2-4f3e-b23a-923bbe2b8093']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092925063', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFPH', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFPH", "wwn-0x5000c50092925063"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFPH-part1", "wwn-0x5000c50092925063-part1"], "labels": [], "masters": [], "uuids": ["986a1d3c-fee2-4f3e-b23a-923bbe2b8093"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFPH", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092925063"}}, "skip_reason": 2020-07-17T11:59:14.869 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-17T11:59:14.966 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF02p8FVVEwc5kVPehvyBSsg4qtsycCzc8e'], 'uuids': ['d6c428ea-29ce-4885-ac43-e98de7b84611']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.945992", "end": "2020-07-17 11:59:14.522757", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF02p8FVVEwc5kVPehvyBSsg4qtsycCzc8e"], "labels": [], "masters": [], "uuids": ["d6c428ea-29ce-4885-ac43-e98de7b84611"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:12.576765", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:14.978 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:14.991 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:16.174 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-so5Nms-Gn41-j0zp-blfS-CLRc-doot-h2dqCf', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300GB400BGN', 'nvme-nvme.8086-43564654363233333030474234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.177383", "end": "2020-07-17 11:59:16.168953", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-so5Nms-Gn41-j0zp-blfS-CLRc-doot-h2dqCf", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GB400BGN", "nvme-nvme.8086-43564654363233333030474234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:14.991570", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-07-17T11:59:16.176 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:16.199 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:16.227 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:16.238 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0SUAE1iKELkEy4LmKodsiyuuRUpNKPKi0'], 'uuids': ['7a7839a7-09d7-4ad0-a70d-113883c748e2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.111171", "end": "2020-07-17 11:59:15.788992", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0SUAE1iKELkEy4LmKodsiyuuRUpNKPKi0"], "labels": [], "masters": [], "uuids": ["7a7839a7-09d7-4ad0-a70d-113883c748e2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:14.677821", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:16.255 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS', 'wwn-0x5000c500871a48d9'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'wwn-0x5000c500871a48d9-part1'], 'uuids': ['986a1d3c-fee2-4f3e-b23a-923bbe2b8093']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a48d9', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HRYS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS", "wwn-0x5000c500871a48d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1", "wwn-0x5000c500871a48d9-part1"], "labels": [], "masters": [], "uuids": ["986a1d3c-fee2-4f3e-b23a-923bbe2b8093"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HRYS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a48d9"}}, "skip_reason": 2020-07-17T11:59:16.258 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-17T11:59:17.537 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bfc5b6Gonsydcrl97J6l78LE00eZ0uGGOg'], 'uuids': ['345ed712-9c8e-4d18-bdad-d4f2ab27edbf']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.140570", "end": "2020-07-17 11:59:17.538285", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bfc5b6Gonsydcrl97J6l78LE00eZ0uGGOg"], "labels": [], "masters": [], "uuids": ["345ed712-9c8e-4d18-bdad-d4f2ab27edbf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:16.397715", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:17.560 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:17.564 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-V7eG1z-btQK-fWSh-irDi-8gUY-fCeR-bv2xCK', 'nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN', 'nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.153820", "end": "2020-07-17 11:59:17.117067", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-V7eG1z-btQK-fWSh-irDi-8gUY-fCeR-bv2xCK", "nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN", "nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:15.963247", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2020-07-17T11:59:17.566 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:17.588 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:17.606 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:18.880 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0DOoljDzDz4hRpmo0fkGtf1n3bYQpZBEM'], 'uuids': ['328497b0-a3ff-4f91-a667-bc2809b583be']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.143062", "end": "2020-07-17 11:59:18.440948", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0DOoljDzDz4hRpmo0fkGtf1n3bYQpZBEM"], "labels": [], "masters": [], "uuids": ["328497b0-a3ff-4f91-a667-bc2809b583be"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:17.297886", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:18.893 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:18.910 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:19.020 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bf1gYoKe5cKoSkkdp16ephYYOwxWJAH1w1'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.303236", "end": "2020-07-17 11:59:19.005078", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bf1gYoKe5cKoSkkdp16ephYYOwxWJAH1w1"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:17.701842", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:19.032 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:20.180 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0somLxoY0vr14bsaVX8fZHRvFjP6hpn5g'], 'uuids': ['9b9755ae-74fb-40c6-ae1e-51853630d16a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.123399", "end": "2020-07-17 11:59:19.725289", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0somLxoY0vr14bsaVX8fZHRvFjP6hpn5g"], "labels": [], "masters": [], "uuids": ["9b9755ae-74fb-40c6-ae1e-51853630d16a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:18.601890", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:20.181 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:20.299 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bfupz6hjDLkxFLq4Sdycb5fGwuVIjqUe8x'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.137270", "end": "2020-07-17 11:59:20.288897", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bfupz6hjDLkxFLq4Sdycb5fGwuVIjqUe8x"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:19.151627", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:20.374 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:20 +0000 (0:00:07.542) 0:01:22.636 *********** 2020-07-17T11:59:20.433 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-07-17T11:59:20.434 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:20.783 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2BfuFHOiZ83arGugHODhQIVsj3AATeCpFuF'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.213835", "end": "2020-07-17 11:59:20.771890", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2BfuFHOiZ83arGugHODhQIVsj3AATeCpFuF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:20.558055", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144903 s, 796 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144903 s, 796 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:20.795 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:20.806 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:21.090 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bf0mYq5l9ctiJdxZoUVyJnonj2wTuVgg08'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.157759", "end": "2020-07-17 11:59:21.088877", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bf0mYq5l9ctiJdxZoUVyJnonj2wTuVgg08"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:20.931118", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154673 s, 746 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154673 s, 746 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:21.106 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFPH', 'wwn-0x5000c50092925063'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFPH-part1', 'wwn-0x5000c50092925063-part1'], 'uuids': ['986a1d3c-fee2-4f3e-b23a-923bbe2b8093']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092925063', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFPH', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFPH", "wwn-0x5000c50092925063"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFPH-part1", "wwn-0x5000c50092925063-part1"], "labels": [], "masters": [], "uuids": ["986a1d3c-fee2-4f3e-b23a-923bbe2b8093"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFPH", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092925063"}}, "skip_reason": 2020-07-17T11:59:21.107 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-17T11:59:21.290 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-so5Nms-Gn41-j0zp-blfS-CLRc-doot-h2dqCf', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300GB400BGN', 'nvme-nvme.8086-43564654363233333030474234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.055667", "end": "2020-07-17 11:59:21.287659", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-so5Nms-Gn41-j0zp-blfS-CLRc-doot-h2dqCf", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GB400BGN", "nvme-nvme.8086-43564654363233333030474234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:21.231992", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0507739 s, 2.3 GB/s", "std 2020-07-17T11:59:21.292 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0507739 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:21.302 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:21.314 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:21.431 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-07-17T11:59:21.433 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0flQf8dOB5O3QkoR8hwqJy65Xzul0Z9vw'], 'uuids': ['46d61db7-8f0b-4a22-a2e6-74b8b76e414c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.124176", "end": "2020-07-17 11:59:20.992978", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0flQf8dOB5O3QkoR8hwqJy65Xzul0Z9vw"], "labels": [], "masters": [], "uuids": ["46d61db7-8f0b-4a22-a2e6-74b8b76e414c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:19.868802", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-17T11:59:21.509 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:21 +0000 (0:00:01.140) 0:01:23.776 *********** 2020-07-17T11:59:21.565 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-07-17T11:59:21.567 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:21.622 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bfc5b6Gonsydcrl97J6l78LE00eZ0uGGOg'], 'uuids': ['345ed712-9c8e-4d18-bdad-d4f2ab27edbf']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.164666", "end": "2020-07-17 11:59:21.610737", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bfc5b6Gonsydcrl97J6l78LE00eZ0uGGOg"], "labels": [], "masters": [], "uuids": ["345ed712-9c8e-4d18-bdad-d4f2ab27edbf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:21.446071", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.161459 s, 714 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.161459 s, 714 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:21.629 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:21.641 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:21.825 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF02p8FVVEwc5kVPehvyBSsg4qtsycCzc8e'], 'uuids': ['d6c428ea-29ce-4885-ac43-e98de7b84611']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.111260", "end": "2020-07-17 11:59:21.387869", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF02p8FVVEwc5kVPehvyBSsg4qtsycCzc8e"], "labels": [], "masters": [], "uuids": ["d6c428ea-29ce-4885-ac43-e98de7b84611"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:21.276609", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0991576 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0991576 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:21.837 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:21.849 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:21.946 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bf1gYoKe5cKoSkkdp16ephYYOwxWJAH1w1'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156507", "end": "2020-07-17 11:59:21.922518", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bf1gYoKe5cKoSkkdp16ephYYOwxWJAH1w1"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:21.766011", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151827 s, 760 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151827 s, 760 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:21.957 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:22.088 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0SUAE1iKELkEy4LmKodsiyuuRUpNKPKi0'], 'uuids': ['7a7839a7-09d7-4ad0-a70d-113883c748e2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.107718", "end": "2020-07-17 11:59:21.647020", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0SUAE1iKELkEy4LmKodsiyuuRUpNKPKi0"], "labels": [], "masters": [], "uuids": ["7a7839a7-09d7-4ad0-a70d-113883c748e2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:21.539302", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.103195 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.103195 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:22.102 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS', 'wwn-0x5000c500871a48d9'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'wwn-0x5000c500871a48d9-part1'], 'uuids': ['986a1d3c-fee2-4f3e-b23a-923bbe2b8093']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a48d9', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HRYS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS", "wwn-0x5000c500871a48d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1", "wwn-0x5000c500871a48d9-part1"], "labels": [], "masters": [], "uuids": ["986a1d3c-fee2-4f3e-b23a-923bbe2b8093"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HRYS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a48d9"}}, "skip_reason": 2020-07-17T11:59:22.103 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-17T11:59:22.250 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bfupz6hjDLkxFLq4Sdycb5fGwuVIjqUe8x'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.156982", "end": "2020-07-17 11:59:22.232346", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-w8jIqnCHNgdWVaxcA1NNJqKR608ou2Bfupz6hjDLkxFLq4Sdycb5fGwuVIjqUe8x"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:22.075364", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152246 s, 758 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152246 s, 758 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:22.303 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:22 +0000 (0:00:00.793) 0:01:24.570 *********** 2020-07-17T11:59:22.313 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-V7eG1z-btQK-fWSh-irDi-8gUY-fCeR-bv2xCK', 'nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN', 'nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.047862", "end": "2020-07-17 11:59:21.827021", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-V7eG1z-btQK-fWSh-irDi-8gUY-fCeR-bv2xCK", "nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN", "nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:21.779159", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0432123 s, 2.7 GB/s", "stderr_lines": [" 2020-07-17T11:59:22.315 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0432123 s, 2.7 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:22.317 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:22.318 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:22.544 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0DOoljDzDz4hRpmo0fkGtf1n3bYQpZBEM'], 'uuids': ['328497b0-a3ff-4f91-a667-bc2809b583be']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.109100", "end": "2020-07-17 11:59:22.105554", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0DOoljDzDz4hRpmo0fkGtf1n3bYQpZBEM"], "labels": [], "masters": [], "uuids": ["328497b0-a3ff-4f91-a667-bc2809b583be"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:21.996454", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.105622 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.105622 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:22.558 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:22.569 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:22.603 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-07-17T11:59:22.605 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.133199", "end": "2020-07-17 11:59:22.584804", "rc": 0, "start": "2020-07-17 11:59:22.451605", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:22.655 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:22 +0000 (0:00:00.351) 0:01:24.922 *********** 2020-07-17T11:59:22.823 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-07-17T11:59:22.824 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0somLxoY0vr14bsaVX8fZHRvFjP6hpn5g'], 'uuids': ['9b9755ae-74fb-40c6-ae1e-51853630d16a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.108816", "end": "2020-07-17 11:59:22.376386", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0somLxoY0vr14bsaVX8fZHRvFjP6hpn5g"], "labels": [], "masters": [], "uuids": ["9b9755ae-74fb-40c6-ae1e-51853630d16a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:22.267570", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.104648 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.104648 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:22.825 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:22.949 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-07-17T11:59:22.951 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.144710", "end": "2020-07-17 11:59:22.952595", "rc": 0, "start": "2020-07-17 11:59:22.807885", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:22.991 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:22 +0000 (0:00:00.335) 0:01:25.258 *********** 2020-07-17T11:59:23.032 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-07-17T11:59:23.078 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.086) 0:01:25.345 *********** 2020-07-17T11:59:23.087 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-07-17T11:59:23.088 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0flQf8dOB5O3QkoR8hwqJy65Xzul0Z9vw'], 'uuids': ['46d61db7-8f0b-4a22-a2e6-74b8b76e414c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.109942", "end": "2020-07-17 11:59:22.626911", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-abIOyoV5lAiMYfE0sv6zKRp1b88VBgF0flQf8dOB5O3QkoR8hwqJy65Xzul0Z9vw"], "labels": [], "masters": [], "uuids": ["46d61db7-8f0b-4a22-a2e6-74b8b76e414c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-17 11:59:22.516969", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.105326 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.105326 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:23.143 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.064) 0:01:25.410 *********** 2020-07-17T11:59:23.153 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-07-17T11:59:23.154 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:23.207 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.064) 0:01:25.474 *********** 2020-07-17T11:59:23.260 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-07-17T11:59:23.261 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:23.299 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.091) 0:01:25.566 *********** 2020-07-17T11:59:23.329 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-07-17T11:59:23.331 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:23.383 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.084) 0:01:25.650 *********** 2020-07-17T11:59:23.422 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-07-17T11:59:23.424 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:23.459 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.075) 0:01:25.726 *********** 2020-07-17T11:59:23.467 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-07-17T11:59:23.468 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.151202", "end": "2020-07-17 11:59:23.021048", "rc": 0, "start": "2020-07-17 11:59:22.869846", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:23.510 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.051) 0:01:25.777 *********** 2020-07-17T11:59:23.601 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-07-17T11:59:23.603 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-17T11:59:23.657 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.146) 0:01:25.924 *********** 2020-07-17T11:59:23.709 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-07-17T11:59:23.711 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'block_used': 13636014, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226459094, 'size_available': 927576449024, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113847, 'block_size': 4096, 'inode_available': 60941129}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226459094, "block_size": 4096, "block_total": 240095108, "block_used": 13636014, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941129, "inode_total": 61054976, "inode_used": 113847, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927576449024, "size_total": 983429562368, "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:23.760 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.102) 0:01:26.026 *********** 2020-07-17T11:59:23.769 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-07-17T11:59:23.771 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.071201", "end": "2020-07-17 11:59:23.306706", "rc": 0, "start": "2020-07-17 11:59:23.235505", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:23.813 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.050) 0:01:26.077 *********** 2020-07-17T11:59:23.821 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-07-17T11:59:23.823 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:23.875 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.064) 0:01:26.142 *********** 2020-07-17T11:59:23.886 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-07-17T11:59:23.932 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.056) 0:01:26.199 *********** 2020-07-17T11:59:23.942 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:23.980 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:23 +0000 (0:00:00.048) 0:01:26.247 *********** 2020-07-17T11:59:23.987 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-07-17T11:59:23.989 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.045 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.062) 0:01:26.310 *********** 2020-07-17T11:59:24.051 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:59:24.053 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.107 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.064) 0:01:26.374 *********** 2020-07-17T11:59:24.121 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.166 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.054) 0:01:26.429 *********** 2020-07-17T11:59:24.172 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-07-17T11:59:24.174 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.228 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.066) 0:01:26.495 *********** 2020-07-17T11:59:24.242 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.284 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.056) 0:01:26.551 *********** 2020-07-17T11:59:24.298 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.347 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.061) 0:01:26.613 *********** 2020-07-17T11:59:24.356 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-07-17T11:59:24.357 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.405 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.058) 0:01:26.672 *********** 2020-07-17T11:59:24.416 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.459 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.054) 0:01:26.726 *********** 2020-07-17T11:59:24.549 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-07-17T11:59:24.551 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-17T11:59:24.619 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.159) 0:01:26.886 *********** 2020-07-17T11:59:24.673 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-07-17T11:59:24.674 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'block_used': 13636026, 'uuid': '986a1d3c-fee2-4f3e-b23a-923bbe2b8093', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226459082, 'size_available': 927576399872, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113846, 'block_size': 4096, 'inode_available': 60941130}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226459082, "block_size": 4096, "block_total": 240095108, "block_used": 13636026, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941130, "inode_total": 61054976, "inode_used": 113846, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927576399872, "size_total": 983429562368, "uuid": "986a1d3c-fee2-4f3e-b23a-923bbe2b8093"}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.716 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.096) 0:01:26.983 *********** 2020-07-17T11:59:24.757 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.794 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.078) 0:01:27.061 *********** 2020-07-17T11:59:24.796 INFO:teuthology.task.ansible.out: 2020-07-17T11:59:24.833 INFO:teuthology.task.ansible.out: 2020-07-17T11:59:24.835 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.882 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.087) 0:01:27.149 *********** 2020-07-17T11:59:24.883 INFO:teuthology.task.ansible.out: 2020-07-17T11:59:24.926 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:59:24.927 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:24.963 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:24 +0000 (0:00:00.081) 0:01:27.230 *********** 2020-07-17T11:59:25.014 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-07-17T11:59:25.016 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:25.058 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:25 +0000 (0:00:00.093) 0:01:27.324 *********** 2020-07-17T11:59:25.097 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-07-17T11:59:25.099 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:25.142 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:25 +0000 (0:00:00.085) 0:01:27.409 *********** 2020-07-17T11:59:25.150 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-07-17T11:59:25.152 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-07-17T11:59:25.192 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:25 +0000 (0:00:00.049) 0:01:27.459 *********** 2020-07-17T11:59:25.203 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-07-17T11:59:25.204 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:25.247 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:25 +0000 (0:00:00.054) 0:01:27.513 *********** 2020-07-17T11:59:25.607 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-07-17T11:59:25.609 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-07-17T11:59:25.674 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:25 +0000 (0:00:00.423) 0:01:27.937 *********** 2020-07-17T11:59:25.821 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-07-17T11:59:25.823 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:25.978 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:26.084 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:26.265 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:26.367 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:26.538 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:26.620 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:26.764 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:26.864 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:26.918 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:26 +0000 (0:00:01.248) 0:01:29.185 *********** 2020-07-17T11:59:27.023 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-17T11:59:27.069 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:27 +0000 (0:00:00.150) 0:01:29.336 *********** 2020-07-17T11:59:27.081 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-07-17T11:59:27.083 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-07-17T11:59:27.134 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:27 +0000 (0:00:00.064) 0:01:29.401 *********** 2020-07-17T11:59:27.231 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-07-17T11:59:27.285 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:27 +0000 (0:00:00.150) 0:01:29.552 *********** 2020-07-17T11:59:27.312 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-07-17T11:59:27.314 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-17T11:59:27.427 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-17T11:59:27.481 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-17T11:59:27.566 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-17T11:59:27.610 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-17T11:59:27.679 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-17T11:59:27.691 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:27.741 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:27 +0000 (0:00:00.455) 0:01:30.007 *********** 2020-07-17T11:59:27.752 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-17T11:59:27.865 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-17T11:59:27.876 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:27.919 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:27 +0000 (0:00:00.178) 0:01:30.186 *********** 2020-07-17T11:59:27.934 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-07-17T11:59:27.936 INFO:teuthology.task.ansible.out: ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-07-17T11:59:28.000 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:27 +0000 (0:00:00.080) 0:01:30.266 *********** 2020-07-17T11:59:28.052 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-07-17T11:59:28.054 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-07-17T11:59:28.110 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:28 +0000 (0:00:00.110) 0:01:30.377 *********** 2020-07-17T11:59:28.126 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-07-17T11:59:28.187 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:28 +0000 (0:00:00.076) 0:01:30.454 *********** 2020-07-17T11:59:28.252 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-07-17T11:59:28.254 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-07-17T11:59:28.294 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:28 +0000 (0:00:00.107) 0:01:30.561 *********** 2020-07-17T11:59:28.605 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-07-17T11:59:28.607 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true} 2020-07-17T11:59:28.650 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:28 +0000 (0:00:00.355) 0:01:30.916 *********** 2020-07-17T11:59:28.668 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true} 2020-07-17T11:59:28.712 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:28 +0000 (0:00:00.062) 0:01:30.979 *********** 2020-07-17T11:59:29.165 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-07-17T11:59:29.225 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:29 +0000 (0:00:00.512) 0:01:31.491 *********** 2020-07-17T11:59:29.234 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-07-17T11:59:29.282 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:29 +0000 (0:00:00.056) 0:01:31.548 *********** 2020-07-17T11:59:29.392 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-07-17T11:59:29.394 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-07-17T11:59:29.434 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:29 +0000 (0:00:00.152) 0:01:31.700 *********** 2020-07-17T11:59:29.450 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-07-17T11:59:29.494 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:29 +0000 (0:00:00.060) 0:01:31.761 *********** 2020-07-17T11:59:29.610 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-07-17T11:59:29.612 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-07-17T11:59:29.649 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:29 +0000 (0:00:00.154) 0:01:31.916 *********** 2020-07-17T11:59:29.657 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-07-17T11:59:29.700 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:29 +0000 (0:00:00.051) 0:01:31.967 *********** 2020-07-17T11:59:30.063 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-07-17T11:59:30.065 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1224, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-07-17T11:59:30.129 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:30 +0000 (0:00:00.428) 0:01:32.395 *********** 2020-07-17T11:59:30.138 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1224, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-07-17T11:59:30.185 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:30 +0000 (0:00:00.056) 0:01:32.452 *********** 2020-07-17T11:59:30.519 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-07-17T11:59:30.521 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-07-17T11:59:30.558 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:30 +0000 (0:00:00.372) 0:01:32.825 *********** 2020-07-17T11:59:30.568 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-07-17T11:59:30.628 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:30 +0000 (0:00:00.069) 0:01:32.895 *********** 2020-07-17T11:59:31.491 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:59:31.532 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:31 +0000 (0:00:00.904) 0:01:33.799 *********** 2020-07-17T11:59:31.533 INFO:teuthology.task.ansible.out: 2020-07-17T11:59:31.555 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:59:31.556 INFO:teuthology.task.ansible.out: 2020-07-17T11:59:31.620 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:31 +0000 (0:00:00.087) 0:01:33.886 *********** 2020-07-17T11:59:31.747 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-07-17T11:59:31.749 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046365", "end": "2020-07-17 11:59:31.320287", "rc": 0, "start": "2020-07-17 11:59:31.273922", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-07-17T11:59:31.797 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:31 +0000 (0:00:00.177) 0:01:34.064 *********** 2020-07-17T11:59:31.825 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047801", "end": "2020-07-17 11:59:31.816679", "rc": 0, "start": "2020-07-17 11:59:31.768878", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-07-17T11:59:31.881 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:31 +0000 (0:00:00.083) 0:01:34.148 *********** 2020-07-17T11:59:31.890 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-07-17T11:59:31.892 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:31.929 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:31 +0000 (0:00:00.048) 0:01:34.196 *********** 2020-07-17T11:59:31.940 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:31.978 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:31 +0000 (0:00:00.048) 0:01:34.245 *********** 2020-07-17T11:59:31.987 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-07-17T11:59:31.989 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:32.034 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:32 +0000 (0:00:00.056) 0:01:34.301 *********** 2020-07-17T11:59:32.047 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:32.091 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:32 +0000 (0:00:00.056) 0:01:34.358 *********** 2020-07-17T11:59:32.924 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-07-17T11:59:32.925 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:59:32.961 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:32 +0000 (0:00:00.870) 0:01:35.228 *********** 2020-07-17T11:59:32.970 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:59:33.008 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.046) 0:01:35.275 *********** 2020-07-17T11:59:33.016 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-07-17T11:59:33.018 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:33.057 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.048) 0:01:35.324 *********** 2020-07-17T11:59:33.065 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:33.105 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.048) 0:01:35.372 *********** 2020-07-17T11:59:33.403 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-07-17T11:59:33.405 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-07-17T11:59:33.459 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.353) 0:01:35.725 *********** 2020-07-17T11:59:33.468 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-07-17T11:59:33.508 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.049) 0:01:35.775 *********** 2020-07-17T11:59:33.517 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-07-17T11:59:33.519 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:33.557 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.048) 0:01:35.824 *********** 2020-07-17T11:59:33.568 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:33.621 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.060) 0:01:35.885 *********** 2020-07-17T11:59:33.800 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-07-17T11:59:33.801 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-07-17 11:56:30 UTC", "ActiveEnterTimestampMonotonic": "23974452", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice systemd-tmpfiles-setup.service basic.target network.target systemd-journald.socket sysinit.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-07-17 11:56:24 UTC", "AssertTimestampMonotonic": "17472064", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-07-17 11:56:24 UTC", "ConditionTimestampMonotonic": "17472064", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-07-17T11:59:33.803 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2027", "ExecMainStartTimestamp": "Fri 2020-07-17 11:56:30 UTC", "ExecMainStartTimestampMonotonic": "23974392", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-07-17 11:56:24 UTC", "InactiveExitTimestampMonotonic": "17472838", "InvocationID": "afa3cb7da21b4b868aa6b920ba57a7fd", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-07-17T11:59:33.804 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2027", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-07-17 11:56:30 UTC", "S 2020-07-17T11:59:33.806 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23974452", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-07-17 11:56:30 UTC", "WatchdogTimestampMonotonic": "23974448", "WatchdogUSec": "0"}} 2020-07-17T11:59:33.843 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.224) 0:01:36.110 *********** 2020-07-17T11:59:33.853 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-07-17 11:57:04 UTC", "ActiveEnterTimestampMonotonic": "17420421", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice systemd-tmpfiles-setup.service basic.target -.mount network.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-07-17 11:57:04 UTC", "AssertTimestampMonotonic": "17312069", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-07-17 11:57:04 UTC", "ConditionTimestampMonotonic": "17312069", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-07-17T11:59:33.861 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1073", "ExecMainStartTimestamp": "Fri 2020-07-17 11:57:04 UTC", "ExecMainStartTimestampMonotonic": "17420412", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-07-17 11:57:04 UTC", "InactiveExitTimestampMonotonic": "17312809", "InvocationID": "25b8fc8957d44926a8ab7b95ddcbbaa0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-07-17T11:59:33.863 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1073", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-07-17 11:57:04 UTC", "S 2020-07-17T11:59:33.865 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "17420421", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-07-17 11:57:04 UTC", "WatchdogTimestampMonotonic": "17420420", "WatchdogUSec": "0"}} 2020-07-17T11:59:33.912 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.068) 0:01:36.178 *********** 2020-07-17T11:59:33.921 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:59:33.923 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-07-17T11:59:33.971 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:33 +0000 (0:00:00.059) 0:01:36.238 *********** 2020-07-17T11:59:33.984 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-07-17T11:59:34.024 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:34 +0000 (0:00:00.052) 0:01:36.291 *********** 2020-07-17T11:59:34.034 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:59:34.036 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:34.088 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:34 +0000 (0:00:00.064) 0:01:36.355 *********** 2020-07-17T11:59:34.097 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:34.143 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:34 +0000 (0:00:00.054) 0:01:36.409 *********** 2020-07-17T11:59:34.152 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-07-17T11:59:34.153 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:34.203 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:34 +0000 (0:00:00.060) 0:01:36.469 *********** 2020-07-17T11:59:34.215 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:34.278 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:34 +0000 (0:00:00.069) 0:01:36.539 *********** 2020-07-17T11:59:35.109 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-07-17T11:59:35.111 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1594987069, "cache_updated": false, "changed": false} 2020-07-17T11:59:35.171 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:35 +0000 (0:00:00.899) 0:01:37.438 *********** 2020-07-17T11:59:35.183 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1594987065, "cache_updated": false, "changed": false} 2020-07-17T11:59:35.235 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:35 +0000 (0:00:00.064) 0:01:37.502 *********** 2020-07-17T11:59:35.248 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-07-17T11:59:35.250 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:35.315 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:35 +0000 (0:00:00.079) 0:01:37.581 *********** 2020-07-17T11:59:35.324 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:35.378 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:35 +0000 (0:00:00.062) 0:01:37.644 *********** 2020-07-17T11:59:35.466 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-07-17T11:59:35.467 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-07-17T11:59:35.544 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:35 +0000 (0:00:00.166) 0:01:37.810 *********** 2020-07-17T11:59:35.554 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-07-17T11:59:35.594 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:35 +0000 (0:00:00.050) 0:01:37.861 *********** 2020-07-17T11:59:35.938 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-07-17T11:59:35.942 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1224, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-07-17T11:59:35.976 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:35 +0000 (0:00:00.381) 0:01:38.242 *********** 2020-07-17T11:59:36.050 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1224, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-07-17T11:59:36.093 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:36 +0000 (0:00:00.116) 0:01:38.359 *********** 2020-07-17T11:59:37.125 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-07-17T11:59:37.160 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:37 +0000 (0:00:01.066) 0:01:39.426 *********** 2020-07-17T11:59:37.173 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-07-17T11:59:37.215 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:37 +0000 (0:00:00.055) 0:01:39.482 *********** 2020-07-17T11:59:37.730 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-07-17T11:59:37.732 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false} ok: [smithi052.front.sepia.ceph.com] => {"changed": false} 2020-07-17T11:59:37.793 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:37 +0000 (0:00:00.577) 0:01:40.060 *********** 2020-07-17T11:59:37.842 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:37 +0000 (0:00:00.048) 0:01:40.109 *********** 2020-07-17T11:59:37.854 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-07-17T11:59:37.856 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-07-17T11:59:37.893 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:37 +0000 (0:00:00.051) 0:01:40.160 *********** 2020-07-17T11:59:37.902 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-07-17T11:59:37.951 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:37 +0000 (0:00:00.058) 0:01:40.218 *********** 2020-07-17T11:59:38.065 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-07-17T11:59:38.067 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1594987001.2533734, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1594986999.7526238, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 10066, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1594986999.7526238, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "761277077", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-07-17T11:59:38.119 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:38 +0000 (0:00:00.167) 0:01:40.386 *********** 2020-07-17T11:59:38.156 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1594987045.2080698, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1594987045.2040696, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 10066, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1594987045.2040696, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "761277077", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-07-17T11:59:38.200 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:38 +0000 (0:00:00.081) 0:01:40.467 *********** 2020-07-17T11:59:38.648 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-07-17T11:59:38.650 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-07-17T11:59:38.718 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:38 +0000 (0:00:00.517) 0:01:40.985 *********** 2020-07-17T11:59:38.728 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-07-17T11:59:38.790 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:38 +0000 (0:00:00.071) 0:01:41.056 *********** 2020-07-17T11:59:38.800 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-07-17T11:59:38.802 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-07-17T11:59:38.855 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:38 +0000 (0:00:00.065) 0:01:41.122 *********** 2020-07-17T11:59:38.868 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-07-17T11:59:38.915 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:38 +0000 (0:00:00.059) 0:01:41.182 *********** 2020-07-17T11:59:38.926 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:38.968 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:38 +0000 (0:00:00.053) 0:01:41.235 *********** 2020-07-17T11:59:38.976 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.017 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.046) 0:01:41.282 *********** 2020-07-17T11:59:39.110 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-07-17T11:59:39.113 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-07-17T11:59:39.165 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.144) 0:01:41.426 *********** 2020-07-17T11:59:39.181 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-07-17T11:59:39.221 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.061) 0:01:41.488 *********** 2020-07-17T11:59:39.326 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-07-17T11:59:39.327 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-07-17T11:59:39.391 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.169) 0:01:41.658 *********** 2020-07-17T11:59:39.405 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-07-17T11:59:39.461 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.070) 0:01:41.728 *********** 2020-07-17T11:59:39.472 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-07-17T11:59:39.474 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.530 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.067) 0:01:41.795 *********** 2020-07-17T11:59:39.538 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.591 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.061) 0:01:41.857 *********** 2020-07-17T11:59:39.614 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-07-17T11:59:39.615 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.617 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.639 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.651 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.653 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.709 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.117) 0:01:41.975 *********** 2020-07-17T11:59:39.719 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.721 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.775 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.064) 0:01:42.039 *********** 2020-07-17T11:59:39.794 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-07-17T11:59:39.796 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.798 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.848 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.075) 0:01:42.115 *********** 2020-07-17T11:59:39.865 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.866 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.880 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.882 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.895 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-17T11:59:39.897 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:39.949 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.100) 0:01:42.216 *********** 2020-07-17T11:59:39.997 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:39 +0000 (0:00:00.048) 0:01:42.264 *********** 2020-07-17T11:59:40.006 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.050 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.053) 0:01:42.317 *********** 2020-07-17T11:59:40.061 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-07-17T11:59:40.063 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.100 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.049) 0:01:42.366 *********** 2020-07-17T11:59:40.109 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.163 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.062) 0:01:42.429 *********** 2020-07-17T11:59:40.176 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-07-17T11:59:40.178 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.239 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.076) 0:01:42.506 *********** 2020-07-17T11:59:40.249 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.308 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.068) 0:01:42.575 *********** 2020-07-17T11:59:40.318 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-07-17T11:59:40.320 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.364 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.055) 0:01:42.630 *********** 2020-07-17T11:59:40.377 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.443 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.079) 0:01:42.710 *********** 2020-07-17T11:59:40.452 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-07-17T11:59:40.455 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.498 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.054) 0:01:42.765 *********** 2020-07-17T11:59:40.507 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.556 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.058) 0:01:42.823 *********** 2020-07-17T11:59:40.567 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-07-17T11:59:40.569 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.723 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.166) 0:01:42.990 *********** 2020-07-17T11:59:40.733 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.774 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.051) 0:01:43.041 *********** 2020-07-17T11:59:40.786 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-07-17T11:59:40.787 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.846 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.072) 0:01:43.113 *********** 2020-07-17T11:59:40.855 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.896 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.049) 0:01:43.163 *********** 2020-07-17T11:59:40.908 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-07-17T11:59:40.910 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:40.955 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:40 +0000 (0:00:00.058) 0:01:43.222 *********** 2020-07-17T11:59:40.963 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-17T11:59:41.014 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:41 +0000 (0:00:00.059) 0:01:43.281 *********** 2020-07-17T11:59:41.131 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-07-17T11:59:41.133 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011007", "end": "2020-07-17 11:59:41.123103", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-07-17 11:59:41.112096", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:41.187 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011001", "end": "2020-07-17 11:59:40.745492", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-07-17 11:59:40.734491", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:41.268 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006116", "end": "2020-07-17 11:59:41.262202", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-07-17 11:59:41.256086", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:41.334 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006110", "end": "2020-07-17 11:59:40.886731", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-07-17 11:59:40.880621", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:41.410 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005333", "end": "2020-07-17 11:59:41.397935", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-07-17 11:59:41.392602", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:41.478 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005628", "end": "2020-07-17 11:59:41.030987", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-07-17 11:59:41.025359", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:41.645 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.030968", "end": "2020-07-17 11:59:41.211717", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-07-17 11:59:41.180749", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:41.647 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.090589", "end": "2020-07-17 11:59:41.636089", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-07-17 11:59:41.545500", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-17T11:59:41.688 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:41 +0000 (0:00:00.673) 0:01:43.954 *********** 2020-07-17T11:59:41.748 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:41 +0000 (0:00:00.059) 0:01:44.014 *********** 2020-07-17T11:59:41.757 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-07-17T11:59:41.808 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:41 +0000 (0:00:00.060) 0:01:44.075 *********** 2020-07-17T11:59:41.867 INFO:teuthology.task.ansible.out:Friday 17 July 2020 11:59:41 +0000 (0:00:00.058) 0:01:44.134 *********** 2020-07-17T11:59:41.963 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-07-17T11:59:41.965 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-07-17T11:59:42.064 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-07-17T11:59:42.164 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-07-17T11:59:42.169 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-07-17T11:59:42.173 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-07-17T11:59:42.175 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi052.front.sepia.ceph.com : ok=108 changed=26 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 smithi146.front.sepia.ceph.com : ok=108 changed=26 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 Friday 17 July 2020 11:59:42 +0000 (0:00:00.307) 0:01:44.441 *********** =============================================================================== testnode : Ensure packages are not present. ----------------------------- 9.11s testnode : Install packages --------------------------------------------- 8.02s testnode : Zap all non-root disks --------------------------------------- 7.54s testnode : Update apt cache. -------------------------------------------- 2.88s common : Update apt cache ----------------------------------------------- 1.92s Gathering Facts --------------------------------------------------------- 1.90s testnode : Install apt keys --------------------------------------------- 1.90s testnode : Create logical volume(s) ------------------------------------- 1.25s common : Set the hardware clock ----------------------------------------- 1.14s testnode : Install packages via pip ------------------------------------- 1.07s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.91s testnode : Install python-pip on apt based systems. --------------------- 0.90s testnode : Install ntp package on apt based systems. -------------------- 0.87s common : Make sure smartmontools is installed --------------------------- 0.84s Gathering Facts --------------------------------------------------------- 0.84s testnode : Install python-apt ------------------------------------------- 0.82s testnode : Blow away lingering OSD data and FSIDs ----------------------- 0.79s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.78s ansible-managed : Create the ansible user. --- 2020-07-17T11:59:42.177 INFO:teuthology.task.ansible.out:--------------------------- 0.74s testnode : List any leftover Ceph artifacts from previous jobs ---------- 0.67s 2020-07-17T11:59:42.503 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi146.front.sepia.ceph.com'), Remote(name='ubuntu@smithi052.front.sepia.ceph.com')] 2020-07-17T11:59:42.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-17T11:59:42.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:59:42.670 INFO:teuthology.orchestra.run.smithi146:> true 2020-07-17T11:59:42.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-17T11:59:42.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-17T11:59:43.025 INFO:teuthology.orchestra.run.smithi052:> true 2020-07-17T11:59:43.223 INFO:teuthology.run_tasks:Running task clock... 2020-07-17T11:59:44.405 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-07-17T11:59:44.406 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-07-17T11:59:44.407 INFO:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-07-17T11:59:44.459 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-07-17T11:59:44.460 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: Command line: ntpd -gq 2020-07-17T11:59:44.460 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: proto: precision = 0.123 usec (-23) 2020-07-17T11:59:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-17T11:59:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:restrict ::: KOD does nothing without LIMITED. 2020-07-17T11:59:44.462 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-17T11:59:44.463 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: restrict ::: KOD does nothing without LIMITED. 2020-07-17T11:59:44.463 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: Listen and drop on 0 v6wildcard [::]:123 2020-07-17T11:59:44.463 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-07-17T11:59:44.463 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: Listen normally on 2 lo 127.0.0.1:123 2020-07-17T11:59:44.463 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: Listen normally on 3 enp3s0f1 172.21.15.146:123 2020-07-17T11:59:44.464 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: Listen normally on 4 lo [::1]:123 2020-07-17T11:59:44.464 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:76fd%5]:123 2020-07-17T11:59:44.464 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:44 ntpd[6560]: Listening on routing socket on fd #22 for interface updates 2020-07-17T11:59:45.460 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:45 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:45.460 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:45 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:45.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:45 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:45.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:45 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:47.460 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:47 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:47.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:47 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:47.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:47 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:47.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:47 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:47.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:47 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:47.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:47 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:47.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:47 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:47.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:47 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:48.460 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:48 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:48.460 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:48 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:48.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:48 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:48.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:48 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:49.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:49.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:49.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:49.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:49.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:49.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:49.463 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:49.463 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:49.463 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:49.463 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:49.464 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:49.464 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:49 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:50.460 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:50 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:50.460 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:50 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:50.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:50 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:50.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:50 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:51.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:51.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:51.461 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:51.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:51.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:51.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:51.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:51.462 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:51.463 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-07-17T11:59:51.463 INFO:teuthology.orchestra.run.smithi146.stderr:17 Jul 11:59:51 ntpd[6560]: can't open /var/log/ntpstats/loopstats.20200717: Permission denied 2020-07-17T11:59:51.463 INFO:teuthology.orchestra.run.smithi146.stdout:17 Jul 11:59:51 ntpd[6560]: ntpd: time set -0.030143 s 2020-07-17T11:59:51.463 INFO:teuthology.orchestra.run.smithi146.stdout:ntpd: time set -0.030143s 2020-07-17T11:59:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2020-07-17T11:59:51.591 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2020-07-17T11:59:51.591 INFO:teuthology.orchestra.run.smithi146.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-17T11:59:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-17T11:59:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-17T11:59:51.592 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-17T11:59:51.592 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-07-17T11:59:51.593 INFO:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-07-17T11:59:51.664 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-07-17T11:59:51.664 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: Command line: ntpd -gq 2020-07-17T11:59:51.665 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: proto: precision = 0.126 usec (-23) 2020-07-17T11:59:51.666 INFO:teuthology.orchestra.run.smithi052.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-17T11:59:51.666 INFO:teuthology.orchestra.run.smithi052.stderr:restrict ::: KOD does nothing without LIMITED. 2020-07-17T11:59:51.666 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-17T11:59:51.666 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: restrict ::: KOD does nothing without LIMITED. 2020-07-17T11:59:51.667 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: Listen and drop on 0 v6wildcard [::]:123 2020-07-17T11:59:51.667 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-07-17T11:59:51.667 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: Listen normally on 2 lo 127.0.0.1:123 2020-07-17T11:59:51.668 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: Listen normally on 3 ens1f0 172.21.15.52:123 2020-07-17T11:59:51.668 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: Listen normally on 4 lo [::1]:123 2020-07-17T11:59:51.668 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15de%3]:123 2020-07-17T11:59:51.668 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:51 ntpd[6908]: Listening on routing socket on fd #22 for interface updates 2020-07-17T11:59:52.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:52 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:52.665 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:52 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:52.665 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:52 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:52.665 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:52 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:52.666 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:52 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:52.666 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:52 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:52.666 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:52 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:52.666 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:52 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:54.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:54 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:54.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:54 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:54.665 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:54 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:54.665 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:54 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:54.665 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:54 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:54.666 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:54 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:54.666 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:54 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:54.666 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:54 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:55.662 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:55 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:55.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:55 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:55.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:55 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:55.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:55 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:56.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:56.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:56.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:56.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:56.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:56.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:56.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:56.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:56.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:56.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:56.665 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:56.665 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:56 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:57.661 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:57 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:57.662 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:57 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:57.662 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:57 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:57.662 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:57 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:58.662 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:58 ntpd[6908]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-17T11:59:58.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:58 ntpd[6908]: can't open /var/log/ntpstats/rawstats.20200717: Permission denied 2020-07-17T11:59:58.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:58 ntpd[6908]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-17T11:59:58.663 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:58 ntpd[6908]: can't open /var/log/ntpstats/peerstats.20200717: Permission denied 2020-07-17T11:59:58.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:58 ntpd[6908]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-07-17T11:59:58.664 INFO:teuthology.orchestra.run.smithi052.stderr:17 Jul 11:59:58 ntpd[6908]: can't open /var/log/ntpstats/loopstats.20200717: Permission denied 2020-07-17T11:59:58.664 INFO:teuthology.orchestra.run.smithi052.stdout:17 Jul 11:59:58 ntpd[6908]: ntpd: time set +0.383542 s 2020-07-17T11:59:58.665 INFO:teuthology.orchestra.run.smithi052.stdout:ntpd: time set +0.383542s 2020-07-17T11:59:58.718 INFO:teuthology.orchestra.run.smithi052.stdout: remote refid st t when poll reach delay offset jitter 2020-07-17T11:59:58.718 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================== 2020-07-17T11:59:58.719 INFO:teuthology.orchestra.run.smithi052.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-17T11:59:58.719 INFO:teuthology.orchestra.run.smithi052.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-17T11:59:58.719 INFO:teuthology.orchestra.run.smithi052.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-17T11:59:58.719 INFO:teuthology.orchestra.run.smithi052.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-17T11:59:58.720 INFO:teuthology.run_tasks:Running task install... 2020-07-17T11:59:58.731 DEBUG:teuthology.task.install:project ceph 2020-07-17T11:59:58.731 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36'}} 2020-07-17T11:59:58.732 DEBUG:teuthology.task.install:config {'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36'} 2020-07-17T11:59:58.732 INFO:teuthology.task.install:Using flavor: basic 2020-07-17T11:59:58.738 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-07-17T11:59:58.739 INFO:teuthology.task.install:extra packages: [] 2020-07-17T11:59:58.746 INFO:teuthology.orchestra.run.smithi146:> sudo lsb_release -is 2020-07-17T11:59:58.811 INFO:teuthology.orchestra.run.smithi146.stdout:Ubuntu 2020-07-17T11:59:58.817 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-17T11:59:58.817 INFO:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2020-07-17T11:59:58.818 INFO:teuthology.orchestra.run.smithi146:> sudo apt-key list | grep Ceph 2020-07-17T11:59:58.872 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2020-07-17T11:59:58.879 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-17T11:59:58.880 INFO:teuthology.orchestra.run.smithi052:> sudo apt-key list | grep Ceph 2020-07-17T11:59:58.911 INFO:teuthology.orchestra.run.smithi146.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-07-17T11:59:58.969 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-07-17T11:59:58.969 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph.com (release key) 2020-07-17T11:59:58.970 INFO:teuthology.orchestra.run.smithi146:> sudo lsb_release -is 2020-07-17T11:59:58.978 INFO:teuthology.orchestra.run.smithi052.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-07-17T11:59:59.021 INFO:teuthology.orchestra.run.smithi146.stdout:Ubuntu 2020-07-17T11:59:59.026 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-17T11:59:59.027 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-07-17T11:59:59.027 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=4dfb1075abb69d7743ad054aace9aa373509ed36 2020-07-17T11:59:59.038 INFO:teuthology.orchestra.run.smithi052.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-07-17T11:59:59.039 INFO:teuthology.orchestra.run.smithi052.stdout:uid [ unknown] Ceph.com (release key) 2020-07-17T11:59:59.039 INFO:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2020-07-17T11:59:59.089 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2020-07-17T11:59:59.095 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-17T11:59:59.095 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-07-17T11:59:59.096 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=4dfb1075abb69d7743ad054aace9aa373509ed36 2020-07-17T11:59:59.220 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default/ 2020-07-17T11:59:59.220 INFO:teuthology.task.install.deb:Package version is 16.0.0-3281-g4dfb1075abb-1bionic 2020-07-17T11:59:59.273 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default/ 2020-07-17T11:59:59.273 INFO:teuthology.task.install.deb:Package version is 16.0.0-3281-g4dfb1075abb-1bionic 2020-07-17T11:59:59.354 INFO:teuthology.orchestra.run.smithi146:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-07-17T11:59:59.401 INFO:teuthology.orchestra.run.smithi052:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-07-17T11:59:59.407 INFO:teuthology.orchestra.run.smithi146:> sudo apt-get update 2020-07-17T11:59:59.455 INFO:teuthology.orchestra.run.smithi052:> sudo apt-get update 2020-07-17T11:59:59.678 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic InRelease 2020-07-17T11:59:59.701 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-07-17T11:59:59.715 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-17T11:59:59.715 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-07-17T11:59:59.720 INFO:teuthology.orchestra.run.smithi052.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic InRelease 2020-07-17T11:59:59.727 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic Release.gpg 2020-07-17T11:59:59.744 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-07-17T11:59:59.750 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-07-17T11:59:59.751 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.9 kB] 2020-07-17T11:59:59.763 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-17T11:59:59.769 INFO:teuthology.orchestra.run.smithi052.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic Release.gpg 2020-07-17T11:59:59.783 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-07-17T11:59:59.784 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,080 B] 2020-07-17T11:59:59.792 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.9 kB] 2020-07-17T11:59:59.825 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,080 B] 2020-07-17T11:59:59.831 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-07-17T12:00:00.078 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-07-17T12:00:00.094 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-07-17T12:00:00.345 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 279 kB in 1s (378 kB/s) 2020-07-17T12:00:00.427 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 279 kB in 1s (363 kB/s) 2020-07-17T12:00:01.541 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2020-07-17T12:00:01.588 INFO:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-3281-g4dfb1075abb-1bionic cephadm=16.0.0-3281-g4dfb1075abb-1bionic ceph-mds=16.0.0-3281-g4dfb1075abb-1bionic ceph-mgr=16.0.0-3281-g4dfb1075abb-1bionic ceph-common=16.0.0-3281-g4dfb1075abb-1bionic ceph-fuse=16.0.0-3281-g4dfb1075abb-1bionic ceph-test=16.0.0-3281-g4dfb1075abb-1bionic radosgw=16.0.0-3281-g4dfb1075abb-1bionic python3-rados=16.0.0-3281-g4dfb1075abb-1bionic python3-rgw=16.0.0-3281-g4dfb1075abb-1bionic python3-cephfs=16.0.0-3281-g4dfb1075abb-1bionic python3-rbd=16.0.0-3281-g4dfb1075abb-1bionic libcephfs2=16.0.0-3281-g4dfb1075abb-1bionic libcephfs-dev=16.0.0-3281-g4dfb1075abb-1bionic librados2=16.0.0-3281-g4dfb1075abb-1bionic librbd1=16.0.0-3281-g4dfb1075abb-1bionic rbd-fuse=16.0.0-3281-g4dfb1075abb-1bionic 2020-07-17T12:00:01.660 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2020-07-17T12:00:01.669 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2020-07-17T12:00:01.708 INFO:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-3281-g4dfb1075abb-1bionic cephadm=16.0.0-3281-g4dfb1075abb-1bionic ceph-mds=16.0.0-3281-g4dfb1075abb-1bionic ceph-mgr=16.0.0-3281-g4dfb1075abb-1bionic ceph-common=16.0.0-3281-g4dfb1075abb-1bionic ceph-fuse=16.0.0-3281-g4dfb1075abb-1bionic ceph-test=16.0.0-3281-g4dfb1075abb-1bionic radosgw=16.0.0-3281-g4dfb1075abb-1bionic python3-rados=16.0.0-3281-g4dfb1075abb-1bionic python3-rgw=16.0.0-3281-g4dfb1075abb-1bionic python3-cephfs=16.0.0-3281-g4dfb1075abb-1bionic python3-rbd=16.0.0-3281-g4dfb1075abb-1bionic libcephfs2=16.0.0-3281-g4dfb1075abb-1bionic libcephfs-dev=16.0.0-3281-g4dfb1075abb-1bionic librados2=16.0.0-3281-g4dfb1075abb-1bionic librbd1=16.0.0-3281-g4dfb1075abb-1bionic rbd-fuse=16.0.0-3281-g4dfb1075abb-1bionic 2020-07-17T12:00:01.784 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2020-07-17T12:00:01.920 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2020-07-17T12:00:01.921 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2020-07-17T12:00:02.044 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2020-07-17T12:00:02.044 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2020-07-17T12:00:02.080 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2020-07-17T12:00:02.080 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-17T12:00:02.080 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-17T12:00:02.083 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-07-17T12:00:02.083 INFO:teuthology.orchestra.run.smithi146.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-07-17T12:00:02.084 INFO:teuthology.orchestra.run.smithi146.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-07-17T12:00:02.084 INFO:teuthology.orchestra.run.smithi146.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-07-17T12:00:02.084 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-07-17T12:00:02.084 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-17T12:00:02.085 INFO:teuthology.orchestra.run.smithi146.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-17T12:00:02.085 INFO:teuthology.orchestra.run.smithi146.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-17T12:00:02.085 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-17T12:00:02.086 INFO:teuthology.orchestra.run.smithi146.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-17T12:00:02.086 INFO:teuthology.orchestra.run.smithi146.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-17T12:00:02.086 INFO:teuthology.orchestra.run.smithi146.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-17T12:00:02.087 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-17T12:00:02.087 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-17T12:00:02.087 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytest python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-07-17T12:00:02.087 INFO:teuthology.orchestra.run.smithi146.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-07-17T12:00:02.088 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-07-17T12:00:02.088 INFO:teuthology.orchestra.run.smithi146.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-07-17T12:00:02.088 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-07-17T12:00:02.089 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-07-17T12:00:02.089 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2020-07-17T12:00:02.090 INFO:teuthology.orchestra.run.smithi146.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-07-17T12:00:02.090 INFO:teuthology.orchestra.run.smithi146.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-07-17T12:00:02.090 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-07-17T12:00:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-07-17T12:00:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-07-17T12:00:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-07-17T12:00:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-07-17T12:00:02.092 INFO:teuthology.orchestra.run.smithi146.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-07-17T12:00:02.092 INFO:teuthology.orchestra.run.smithi146.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-07-17T12:00:02.092 INFO:teuthology.orchestra.run.smithi146.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-07-17T12:00:02.093 INFO:teuthology.orchestra.run.smithi146.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-07-17T12:00:02.094 INFO:teuthology.orchestra.run.smithi146.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-07-17T12:00:02.095 INFO:teuthology.orchestra.run.smithi146.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-07-17T12:00:02.095 INFO:teuthology.orchestra.run.smithi146.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-07-17T12:00:02.095 INFO:teuthology.orchestra.run.smithi146.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-07-17T12:00:02.183 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2020-07-17T12:00:02.183 INFO:teuthology.orchestra.run.smithi146.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-07-17T12:00:02.183 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-17T12:00:02.184 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-17T12:00:02.185 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-07-17T12:00:02.186 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-07-17T12:00:02.186 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-07-17T12:00:02.186 INFO:teuthology.orchestra.run.smithi146.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-07-17T12:00:02.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-07-17T12:00:02.186 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-07-17T12:00:02.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-17T12:00:02.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-17T12:00:02.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-17T12:00:02.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-17T12:00:02.187 INFO:teuthology.orchestra.run.smithi146.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-17T12:00:02.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-17T12:00:02.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-17T12:00:02.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-17T12:00:02.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-17T12:00:02.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-07-17T12:00:02.188 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-07-17T12:00:02.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-07-17T12:00:02.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-07-17T12:00:02.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-07-17T12:00:02.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-07-17T12:00:02.189 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-07-17T12:00:02.190 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages will be upgraded: 2020-07-17T12:00:02.190 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 librbd1 2020-07-17T12:00:02.226 INFO:teuthology.orchestra.run.smithi052.stdout:The following additional packages will be installed: 2020-07-17T12:00:02.227 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-17T12:00:02.227 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-17T12:00:02.230 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-07-17T12:00:02.230 INFO:teuthology.orchestra.run.smithi052.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-07-17T12:00:02.230 INFO:teuthology.orchestra.run.smithi052.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-07-17T12:00:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-07-17T12:00:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-07-17T12:00:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-17T12:00:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-17T12:00:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-17T12:00:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-17T12:00:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-17T12:00:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-17T12:00:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-17T12:00:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-17T12:00:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-17T12:00:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytest python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-07-17T12:00:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-07-17T12:00:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-07-17T12:00:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-07-17T12:00:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-07-17T12:00:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-07-17T12:00:02.234 INFO:teuthology.orchestra.run.smithi052.stdout:Suggested packages: 2020-07-17T12:00:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-07-17T12:00:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-07-17T12:00:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-07-17T12:00:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-07-17T12:00:02.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-07-17T12:00:02.238 INFO:teuthology.orchestra.run.smithi052.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-07-17T12:00:02.238 INFO:teuthology.orchestra.run.smithi052.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-07-17T12:00:02.238 INFO:teuthology.orchestra.run.smithi052.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-07-17T12:00:02.239 INFO:teuthology.orchestra.run.smithi052.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-07-17T12:00:02.239 INFO:teuthology.orchestra.run.smithi052.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-07-17T12:00:02.239 INFO:teuthology.orchestra.run.smithi052.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-07-17T12:00:02.240 INFO:teuthology.orchestra.run.smithi052.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-07-17T12:00:02.240 INFO:teuthology.orchestra.run.smithi052.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-07-17T12:00:02.240 INFO:teuthology.orchestra.run.smithi052.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-07-17T12:00:02.240 INFO:teuthology.orchestra.run.smithi052.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-07-17T12:00:02.338 INFO:teuthology.orchestra.run.smithi052.stdout:The following NEW packages will be installed: 2020-07-17T12:00:02.338 INFO:teuthology.orchestra.run.smithi052.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-07-17T12:00:02.338 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-17T12:00:02.339 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-17T12:00:02.341 INFO:teuthology.orchestra.run.smithi146.stdout:2 upgraded, 110 newly installed, 0 to remove and 1 not upgraded. 2020-07-17T12:00:02.341 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 164 MB of archives. 2020-07-17T12:00:02.341 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 765 MB of additional disk space will be used. 2020-07-17T12:00:02.342 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-3281-g4dfb1075abb-1bionic [1,679 kB] 2020-07-17T12:00:02.342 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-07-17T12:00:02.342 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-07-17T12:00:02.342 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-07-17T12:00:02.342 INFO:teuthology.orchestra.run.smithi052.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-07-17T12:00:02.343 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-07-17T12:00:02.343 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-07-17T12:00:02.343 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-17T12:00:02.343 INFO:teuthology.orchestra.run.smithi052.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-17T12:00:02.343 INFO:teuthology.orchestra.run.smithi052.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-17T12:00:02.344 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-17T12:00:02.344 INFO:teuthology.orchestra.run.smithi052.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-17T12:00:02.344 INFO:teuthology.orchestra.run.smithi052.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-17T12:00:02.344 INFO:teuthology.orchestra.run.smithi052.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-17T12:00:02.344 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-17T12:00:02.344 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-17T12:00:02.345 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-07-17T12:00:02.345 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-07-17T12:00:02.345 INFO:teuthology.orchestra.run.smithi052.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-07-17T12:00:02.345 INFO:teuthology.orchestra.run.smithi052.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-07-17T12:00:02.345 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-07-17T12:00:02.346 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-07-17T12:00:02.346 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-07-17T12:00:02.347 INFO:teuthology.orchestra.run.smithi052.stdout:The following packages will be upgraded: 2020-07-17T12:00:02.348 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 librbd1 2020-07-17T12:00:02.506 INFO:teuthology.orchestra.run.smithi052.stdout:2 upgraded, 110 newly installed, 0 to remove and 1 not upgraded. 2020-07-17T12:00:02.507 INFO:teuthology.orchestra.run.smithi052.stdout:Need to get 164 MB of archives. 2020-07-17T12:00:02.507 INFO:teuthology.orchestra.run.smithi052.stdout:After this operation, 765 MB of additional disk space will be used. 2020-07-17T12:00:02.507 INFO:teuthology.orchestra.run.smithi052.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-3281-g4dfb1075abb-1bionic [1,679 kB] 2020-07-17T12:00:02.509 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-3281-g4dfb1075abb-1bionic [3,145 kB] 2020-07-17T12:00:02.628 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-3281-g4dfb1075abb-1bionic [460 kB] 2020-07-17T12:00:02.656 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-3281-g4dfb1075abb-1bionic [345 kB] 2020-07-17T12:00:02.673 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-3281-g4dfb1075abb-1bionic [20.6 kB] 2020-07-17T12:00:02.673 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-3281-g4dfb1075abb-1bionic [168 kB] 2020-07-17T12:00:02.680 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-3281-g4dfb1075abb-1bionic [25.6 kB] 2020-07-17T12:00:02.681 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-3281-g4dfb1075abb-1bionic [312 kB] 2020-07-17T12:00:02.702 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-3281-g4dfb1075abb-1bionic [3,004 kB] 2020-07-17T12:00:02.729 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-3281-g4dfb1075abb-1bionic [3,145 kB] 2020-07-17T12:00:02.851 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-3281-g4dfb1075abb-1bionic [99.3 kB] 2020-07-17T12:00:02.853 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-3281-g4dfb1075abb-1bionic [412 kB] 2020-07-17T12:00:02.866 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-3281-g4dfb1075abb-1bionic [19.7 MB] 2020-07-17T12:00:03.004 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-3281-g4dfb1075abb-1bionic [460 kB] 2020-07-17T12:00:03.057 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-3281-g4dfb1075abb-1bionic [345 kB] 2020-07-17T12:00:03.069 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-3281-g4dfb1075abb-1bionic [20.6 kB] 2020-07-17T12:00:03.070 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-3281-g4dfb1075abb-1bionic [168 kB] 2020-07-17T12:00:03.114 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-3281-g4dfb1075abb-1bionic [25.6 kB] 2020-07-17T12:00:03.115 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-3281-g4dfb1075abb-1bionic [312 kB] 2020-07-17T12:00:03.127 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-3281-g4dfb1075abb-1bionic [3,004 kB] 2020-07-17T12:00:03.410 INFO:teuthology.orchestra.run.smithi052.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-3281-g4dfb1075abb-1bionic [99.3 kB] 2020-07-17T12:00:03.414 INFO:teuthology.orchestra.run.smithi052.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-3281-g4dfb1075abb-1bionic [412 kB] 2020-07-17T12:00:03.478 INFO:teuthology.orchestra.run.smithi052.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-3281-g4dfb1075abb-1bionic [19.7 MB] 2020-07-17T12:00:03.898 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-3281-g4dfb1075abb-1bionic [5,557 kB] 2020-07-17T12:00:04.020 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-07-17T12:00:04.169 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-3281-g4dfb1075abb-1bionic [146 kB] 2020-07-17T12:00:04.173 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-3281-g4dfb1075abb-1bionic [1,352 kB] 2020-07-17T12:00:04.189 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-07-17T12:00:04.241 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-3281-g4dfb1075abb-1bionic [6,574 kB] 2020-07-17T12:00:04.437 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-07-17T12:00:04.447 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-07-17T12:00:04.464 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-07-17T12:00:04.555 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-07-17T12:00:04.558 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-07-17T12:00:04.577 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-3281-g4dfb1075abb-1bionic [24.6 MB] 2020-07-17T12:00:04.590 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-07-17T12:00:04.599 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-07-17T12:00:04.613 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-07-17T12:00:04.618 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-07-17T12:00:04.626 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-07-17T12:00:04.647 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-07-17T12:00:04.649 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-07-17T12:00:04.673 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-07-17T12:00:04.685 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-07-17T12:00:04.686 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-07-17T12:00:04.751 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-07-17T12:00:04.752 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-07-17T12:00:04.754 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-07-17T12:00:04.755 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-07-17T12:00:04.756 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-07-17T12:00:04.758 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-07-17T12:00:04.761 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-07-17T12:00:04.830 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-07-17T12:00:04.831 INFO:teuthology.orchestra.run.smithi146.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-07-17T12:00:04.850 INFO:teuthology.orchestra.run.smithi146.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-07-17T12:00:04.908 INFO:teuthology.orchestra.run.smithi146.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-07-17T12:00:04.908 INFO:teuthology.orchestra.run.smithi146.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-07-17T12:00:04.909 INFO:teuthology.orchestra.run.smithi146.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-07-17T12:00:04.913 INFO:teuthology.orchestra.run.smithi146.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-07-17T12:00:04.921 INFO:teuthology.orchestra.run.smithi146.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-07-17T12:00:04.925 INFO:teuthology.orchestra.run.smithi146.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-07-17T12:00:04.926 INFO:teuthology.orchestra.run.smithi146.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-07-17T12:00:04.986 INFO:teuthology.orchestra.run.smithi146.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-07-17T12:00:05.073 INFO:teuthology.orchestra.run.smithi146.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-07-17T12:00:05.074 INFO:teuthology.orchestra.run.smithi146.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-07-17T12:00:05.222 INFO:teuthology.orchestra.run.smithi052.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-3281-g4dfb1075abb-1bionic [5,557 kB] 2020-07-17T12:00:05.341 INFO:teuthology.orchestra.run.smithi146.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-07-17T12:00:05.404 INFO:teuthology.orchestra.run.smithi146.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-07-17T12:00:05.406 INFO:teuthology.orchestra.run.smithi146.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-07-17T12:00:05.423 INFO:teuthology.orchestra.run.smithi146.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-07-17T12:00:05.423 INFO:teuthology.orchestra.run.smithi146.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-07-17T12:00:05.424 INFO:teuthology.orchestra.run.smithi146.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-07-17T12:00:05.425 INFO:teuthology.orchestra.run.smithi146.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-07-17T12:00:05.425 INFO:teuthology.orchestra.run.smithi146.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-07-17T12:00:05.426 INFO:teuthology.orchestra.run.smithi146.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-07-17T12:00:05.426 INFO:teuthology.orchestra.run.smithi146.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-07-17T12:00:05.499 INFO:teuthology.orchestra.run.smithi146.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-07-17T12:00:05.500 INFO:teuthology.orchestra.run.smithi146.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-07-17T12:00:05.577 INFO:teuthology.orchestra.run.smithi146.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-07-17T12:00:05.587 INFO:teuthology.orchestra.run.smithi146.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-07-17T12:00:05.587 INFO:teuthology.orchestra.run.smithi146.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-07-17T12:00:05.588 INFO:teuthology.orchestra.run.smithi146.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-07-17T12:00:05.588 INFO:teuthology.orchestra.run.smithi146.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-07-17T12:00:05.589 INFO:teuthology.orchestra.run.smithi146.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-07-17T12:00:05.590 INFO:teuthology.orchestra.run.smithi146.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-07-17T12:00:05.591 INFO:teuthology.orchestra.run.smithi146.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-07-17T12:00:05.655 INFO:teuthology.orchestra.run.smithi146.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-07-17T12:00:05.658 INFO:teuthology.orchestra.run.smithi146.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-07-17T12:00:05.725 INFO:teuthology.orchestra.run.smithi052.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-3281-g4dfb1075abb-1bionic [146 kB] 2020-07-17T12:00:05.730 INFO:teuthology.orchestra.run.smithi052.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-3281-g4dfb1075abb-1bionic [1,352 kB] 2020-07-17T12:00:05.734 INFO:teuthology.orchestra.run.smithi146.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-07-17T12:00:05.734 INFO:teuthology.orchestra.run.smithi146.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-07-17T12:00:05.735 INFO:teuthology.orchestra.run.smithi146.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-07-17T12:00:05.736 INFO:teuthology.orchestra.run.smithi146.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-07-17T12:00:05.741 INFO:teuthology.orchestra.run.smithi146.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-07-17T12:00:05.742 INFO:teuthology.orchestra.run.smithi146.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-07-17T12:00:05.743 INFO:teuthology.orchestra.run.smithi146.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-07-17T12:00:05.744 INFO:teuthology.orchestra.run.smithi146.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-07-17T12:00:05.817 INFO:teuthology.orchestra.run.smithi146.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-07-17T12:00:05.817 INFO:teuthology.orchestra.run.smithi146.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-07-17T12:00:05.854 INFO:teuthology.orchestra.run.smithi146.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-3281-g4dfb1075abb-1bionic [3,816 B] 2020-07-17T12:00:05.855 INFO:teuthology.orchestra.run.smithi146.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-3281-g4dfb1075abb-1bionic [540 kB] 2020-07-17T12:00:05.868 INFO:teuthology.orchestra.run.smithi052.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-3281-g4dfb1075abb-1bionic [6,574 kB] 2020-07-17T12:00:05.890 INFO:teuthology.orchestra.run.smithi146.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-07-17T12:00:05.892 INFO:teuthology.orchestra.run.smithi146.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-07-17T12:00:05.893 INFO:teuthology.orchestra.run.smithi146.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-07-17T12:00:05.895 INFO:teuthology.orchestra.run.smithi146.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-3281-g4dfb1075abb-1bionic [1,949 kB] 2020-07-17T12:00:05.988 INFO:teuthology.orchestra.run.smithi146.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-3281-g4dfb1075abb-1bionic [43.1 kB] 2020-07-17T12:00:05.989 INFO:teuthology.orchestra.run.smithi146.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-3281-g4dfb1075abb-1bionic [44.3 kB] 2020-07-17T12:00:05.991 INFO:teuthology.orchestra.run.smithi146.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-07-17T12:00:05.991 INFO:teuthology.orchestra.run.smithi146.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-3281-g4dfb1075abb-1bionic [3,120 kB] 2020-07-17T12:00:05.992 INFO:teuthology.orchestra.run.smithi146.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-07-17T12:00:05.993 INFO:teuthology.orchestra.run.smithi146.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.2 [329 kB] 2020-07-17T12:00:05.995 INFO:teuthology.orchestra.run.smithi146.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-07-17T12:00:05.996 INFO:teuthology.orchestra.run.smithi146.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-07-17T12:00:05.997 INFO:teuthology.orchestra.run.smithi146.stdout:Get:101 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-07-17T12:00:05.997 INFO:teuthology.orchestra.run.smithi146.stdout:Get:102 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-07-17T12:00:06.047 INFO:teuthology.orchestra.run.smithi146.stdout:Get:103 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-07-17T12:00:06.048 INFO:teuthology.orchestra.run.smithi146.stdout:Get:104 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-07-17T12:00:06.149 INFO:teuthology.orchestra.run.smithi146.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 16.0.0-3281-g4dfb1075abb-1bionic [34.2 kB] 2020-07-17T12:00:06.155 INFO:teuthology.orchestra.run.smithi146.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-3281-g4dfb1075abb-1bionic [7,881 kB] 2020-07-17T12:00:06.427 INFO:teuthology.orchestra.run.smithi052.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-3281-g4dfb1075abb-1bionic [24.6 MB] 2020-07-17T12:00:06.553 INFO:teuthology.orchestra.run.smithi146.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-3281-g4dfb1075abb-1bionic [3,932 B] 2020-07-17T12:00:06.553 INFO:teuthology.orchestra.run.smithi146.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-3281-g4dfb1075abb-1bionic [22.3 kB] 2020-07-17T12:00:06.554 INFO:teuthology.orchestra.run.smithi146.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-3281-g4dfb1075abb-1bionic [47.4 MB] 2020-07-17T12:00:07.794 INFO:teuthology.orchestra.run.smithi052.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-07-17T12:00:08.221 INFO:teuthology.orchestra.run.smithi052.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-07-17T12:00:08.468 INFO:teuthology.orchestra.run.smithi052.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-07-17T12:00:08.482 INFO:teuthology.orchestra.run.smithi052.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-07-17T12:00:08.496 INFO:teuthology.orchestra.run.smithi052.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-07-17T12:00:08.601 INFO:teuthology.orchestra.run.smithi052.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-07-17T12:00:08.607 INFO:teuthology.orchestra.run.smithi052.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-07-17T12:00:08.637 INFO:teuthology.orchestra.run.smithi052.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-07-17T12:00:08.644 INFO:teuthology.orchestra.run.smithi052.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-07-17T12:00:08.653 INFO:teuthology.orchestra.run.smithi052.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-07-17T12:00:08.664 INFO:teuthology.orchestra.run.smithi052.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-07-17T12:00:08.672 INFO:teuthology.orchestra.run.smithi052.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-07-17T12:00:08.695 INFO:teuthology.orchestra.run.smithi052.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-07-17T12:00:08.699 INFO:teuthology.orchestra.run.smithi052.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-07-17T12:00:08.712 INFO:teuthology.orchestra.run.smithi052.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-07-17T12:00:08.725 INFO:teuthology.orchestra.run.smithi052.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-07-17T12:00:08.793 INFO:teuthology.orchestra.run.smithi052.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-07-17T12:00:08.794 INFO:teuthology.orchestra.run.smithi052.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-07-17T12:00:08.795 INFO:teuthology.orchestra.run.smithi052.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-07-17T12:00:08.798 INFO:teuthology.orchestra.run.smithi052.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-07-17T12:00:08.805 INFO:teuthology.orchestra.run.smithi052.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-07-17T12:00:08.806 INFO:teuthology.orchestra.run.smithi052.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-07-17T12:00:08.806 INFO:teuthology.orchestra.run.smithi052.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-07-17T12:00:08.806 INFO:teuthology.orchestra.run.smithi052.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-07-17T12:00:08.875 INFO:teuthology.orchestra.run.smithi052.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-07-17T12:00:08.875 INFO:teuthology.orchestra.run.smithi052.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-07-17T12:00:08.889 INFO:teuthology.orchestra.run.smithi146.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-3281-g4dfb1075abb-1bionic [19.2 kB] 2020-07-17T12:00:08.890 INFO:teuthology.orchestra.run.smithi146.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-3281-g4dfb1075abb-1bionic [9,134 kB] 2020-07-17T12:00:08.903 INFO:teuthology.orchestra.run.smithi052.stdout:Get:44 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-3281-g4dfb1075abb-1bionic [3,816 B] 2020-07-17T12:00:08.904 INFO:teuthology.orchestra.run.smithi052.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-3281-g4dfb1075abb-1bionic [540 kB] 2020-07-17T12:00:08.918 INFO:teuthology.orchestra.run.smithi052.stdout:Get:46 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-3281-g4dfb1075abb-1bionic [1,949 kB] 2020-07-17T12:00:08.961 INFO:teuthology.orchestra.run.smithi052.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-07-17T12:00:08.961 INFO:teuthology.orchestra.run.smithi052.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-07-17T12:00:08.961 INFO:teuthology.orchestra.run.smithi052.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-07-17T12:00:08.961 INFO:teuthology.orchestra.run.smithi052.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-07-17T12:00:08.962 INFO:teuthology.orchestra.run.smithi052.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-07-17T12:00:08.968 INFO:teuthology.orchestra.run.smithi052.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-07-17T12:00:08.972 INFO:teuthology.orchestra.run.smithi052.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-07-17T12:00:08.973 INFO:teuthology.orchestra.run.smithi052.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-07-17T12:00:09.035 INFO:teuthology.orchestra.run.smithi052.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-07-17T12:00:09.120 INFO:teuthology.orchestra.run.smithi052.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-07-17T12:00:09.120 INFO:teuthology.orchestra.run.smithi052.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-07-17T12:00:09.141 INFO:teuthology.orchestra.run.smithi052.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-3281-g4dfb1075abb-1bionic [43.1 kB] 2020-07-17T12:00:09.142 INFO:teuthology.orchestra.run.smithi052.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-3281-g4dfb1075abb-1bionic [44.3 kB] 2020-07-17T12:00:09.143 INFO:teuthology.orchestra.run.smithi052.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-3281-g4dfb1075abb-1bionic [3,120 kB] 2020-07-17T12:00:09.324 INFO:teuthology.orchestra.run.smithi146.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-3281-g4dfb1075abb-1bionic [76.1 kB] 2020-07-17T12:00:09.374 INFO:teuthology.orchestra.run.smithi052.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 16.0.0-3281-g4dfb1075abb-1bionic [34.2 kB] 2020-07-17T12:00:09.375 INFO:teuthology.orchestra.run.smithi052.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-3281-g4dfb1075abb-1bionic [7,881 kB] 2020-07-17T12:00:09.405 INFO:teuthology.orchestra.run.smithi052.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-07-17T12:00:09.458 INFO:teuthology.orchestra.run.smithi052.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-07-17T12:00:09.460 INFO:teuthology.orchestra.run.smithi052.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-07-17T12:00:09.489 INFO:teuthology.orchestra.run.smithi052.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-07-17T12:00:09.490 INFO:teuthology.orchestra.run.smithi052.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-07-17T12:00:09.490 INFO:teuthology.orchestra.run.smithi052.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-07-17T12:00:09.491 INFO:teuthology.orchestra.run.smithi052.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-07-17T12:00:09.491 INFO:teuthology.orchestra.run.smithi052.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-07-17T12:00:09.492 INFO:teuthology.orchestra.run.smithi052.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-07-17T12:00:09.492 INFO:teuthology.orchestra.run.smithi052.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-07-17T12:00:09.567 INFO:teuthology.orchestra.run.smithi052.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-07-17T12:00:09.569 INFO:teuthology.orchestra.run.smithi052.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-07-17T12:00:09.642 INFO:teuthology.orchestra.run.smithi052.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-3281-g4dfb1075abb-1bionic [3,932 B] 2020-07-17T12:00:09.642 INFO:teuthology.orchestra.run.smithi052.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-3281-g4dfb1075abb-1bionic [22.3 kB] 2020-07-17T12:00:09.643 INFO:teuthology.orchestra.run.smithi052.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-3281-g4dfb1075abb-1bionic [47.4 MB] 2020-07-17T12:00:09.648 INFO:teuthology.orchestra.run.smithi052.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-07-17T12:00:09.650 INFO:teuthology.orchestra.run.smithi052.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-07-17T12:00:09.656 INFO:teuthology.orchestra.run.smithi052.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-07-17T12:00:09.656 INFO:teuthology.orchestra.run.smithi052.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-07-17T12:00:09.657 INFO:teuthology.orchestra.run.smithi052.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-07-17T12:00:09.660 INFO:teuthology.orchestra.run.smithi052.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-07-17T12:00:09.663 INFO:teuthology.orchestra.run.smithi052.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-07-17T12:00:09.663 INFO:teuthology.orchestra.run.smithi052.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-07-17T12:00:09.729 INFO:teuthology.orchestra.run.smithi052.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-07-17T12:00:09.733 INFO:teuthology.orchestra.run.smithi052.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-07-17T12:00:09.809 INFO:teuthology.orchestra.run.smithi052.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-07-17T12:00:09.810 INFO:teuthology.orchestra.run.smithi052.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-07-17T12:00:09.811 INFO:teuthology.orchestra.run.smithi052.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-07-17T12:00:09.811 INFO:teuthology.orchestra.run.smithi052.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-07-17T12:00:09.813 INFO:teuthology.orchestra.run.smithi052.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-07-17T12:00:09.817 INFO:teuthology.orchestra.run.smithi052.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-07-17T12:00:09.818 INFO:teuthology.orchestra.run.smithi052.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-07-17T12:00:09.819 INFO:teuthology.orchestra.run.smithi052.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-07-17T12:00:09.890 INFO:teuthology.orchestra.run.smithi052.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-07-17T12:00:09.892 INFO:teuthology.orchestra.run.smithi052.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-07-17T12:00:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-07-17T12:00:09.972 INFO:teuthology.orchestra.run.smithi052.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-07-17T12:00:09.973 INFO:teuthology.orchestra.run.smithi052.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-07-17T12:00:10.021 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 164 MB in 7s (23.0 MB/s) 2020-07-17T12:00:10.076 INFO:teuthology.orchestra.run.smithi052.stdout:Get:101 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-07-17T12:00:10.077 INFO:teuthology.orchestra.run.smithi052.stdout:Get:102 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-07-17T12:00:10.077 INFO:teuthology.orchestra.run.smithi052.stdout:Get:103 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.2 [329 kB] 2020-07-17T12:00:10.078 INFO:teuthology.orchestra.run.smithi052.stdout:Get:104 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-07-17T12:00:10.078 INFO:teuthology.orchestra.run.smithi052.stdout:Get:105 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-07-17T12:00:10.078 INFO:teuthology.orchestra.run.smithi052.stdout:Get:106 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-07-17T12:00:10.078 INFO:teuthology.orchestra.run.smithi052.stdout:Get:107 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-07-17T12:00:10.131 INFO:teuthology.orchestra.run.smithi052.stdout:Get:108 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-07-17T12:00:10.133 INFO:teuthology.orchestra.run.smithi052.stdout:Get:109 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-07-17T12:00:10.152 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-uritemplate. 2020-07-17T12:00:10.192 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102053 files and directories currently installed.) 2020-07-17T12:00:10.195 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-07-17T12:00:10.237 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-07-17T12:00:10.546 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-07-17T12:00:10.559 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-07-17T12:00:10.629 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-17T12:00:10.994 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../002-librbd1_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:11.112 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librbd1 (16.0.0-3281-g4dfb1075abb-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-17T12:00:11.207 INFO:teuthology.orchestra.run.smithi052.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-3281-g4dfb1075abb-1bionic [19.2 kB] 2020-07-17T12:00:11.207 INFO:teuthology.orchestra.run.smithi052.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-3281-g4dfb1075abb-1bionic [9,134 kB] 2020-07-17T12:00:11.503 INFO:teuthology.orchestra.run.smithi052.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-dang-zipper-6/4dfb1075abb69d7743ad054aace9aa373509ed36/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-3281-g4dfb1075abb-1bionic [76.1 kB] 2020-07-17T12:00:11.624 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../003-librados2_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:11.747 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librados2 (16.0.0-3281-g4dfb1075abb-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-17T12:00:12.212 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 164 MB in 9s (17.9 MB/s) 2020-07-17T12:00:12.359 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-uritemplate. 2020-07-17T12:00:12.381 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs2. 2020-07-17T12:00:12.393 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:12.396 INFO:teuthology.orchestra.run.smithi052.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102053 files and directories currently installed.) 2020-07-17T12:00:12.399 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-07-17T12:00:12.431 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:12.441 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-07-17T12:00:12.718 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rados. 2020-07-17T12:00:12.728 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../005-python3-rados_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:12.759 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rados (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:12.777 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-07-17T12:00:12.790 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-07-17T12:00:12.986 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-17T12:00:13.112 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-07-17T12:00:13.116 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:13.154 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-argparse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:13.327 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../002-librbd1_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:13.449 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librbd1 (16.0.0-3281-g4dfb1075abb-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-17T12:00:13.456 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cephfs. 2020-07-17T12:00:13.468 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:13.506 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cephfs (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:13.825 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-common. 2020-07-17T12:00:13.838 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:13.875 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-common (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:13.958 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../003-librados2_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:14.054 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librados2 (16.0.0-3281-g4dfb1075abb-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-17T12:00:14.161 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-prettytable. 2020-07-17T12:00:14.174 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-07-17T12:00:14.203 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-07-17T12:00:14.456 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rbd. 2020-07-17T12:00:14.466 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:14.505 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rbd (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:14.698 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libcephfs2. 2020-07-17T12:00:14.711 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:14.749 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libcephfs2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:14.875 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-07-17T12:00:14.886 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-07-17T12:00:14.925 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-17T12:00:15.043 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rados. 2020-07-17T12:00:15.056 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../005-python3-rados_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:15.093 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rados (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:15.253 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-07-17T12:00:15.265 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-07-17T12:00:15.302 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-17T12:00:15.412 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-07-17T12:00:15.424 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:15.454 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-ceph-argparse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:15.605 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librgw2. 2020-07-17T12:00:15.617 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../013-librgw2_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:15.655 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librgw2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:15.706 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cephfs. 2020-07-17T12:00:15.717 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:15.765 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cephfs (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:16.117 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-ceph-common. 2020-07-17T12:00:16.129 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:16.170 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-ceph-common (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:16.174 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rgw. 2020-07-17T12:00:16.186 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:16.224 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rgw (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:16.478 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-prettytable. 2020-07-17T12:00:16.491 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-07-17T12:00:16.528 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-07-17T12:00:16.551 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-07-17T12:00:16.564 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-07-17T12:00:16.601 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-17T12:00:16.781 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rbd. 2020-07-17T12:00:16.786 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:16.817 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rbd (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:16.923 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-07-17T12:00:16.935 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-07-17T12:00:16.973 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-07-17T12:00:17.169 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-07-17T12:00:17.181 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-07-17T12:00:17.219 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-17T12:00:17.335 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liboath0. 2020-07-17T12:00:17.346 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-07-17T12:00:17.384 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-07-17T12:00:17.514 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-07-17T12:00:17.527 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-07-17T12:00:17.555 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-17T12:00:17.695 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libradosstriper1. 2020-07-17T12:00:17.707 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:17.745 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libradosstriper1 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:17.835 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package librgw2. 2020-07-17T12:00:17.846 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../013-librgw2_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:17.876 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librgw2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:18.182 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-common. 2020-07-17T12:00:18.193 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../019-ceph-common_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:18.231 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-common (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:18.419 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rgw. 2020-07-17T12:00:18.420 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:18.452 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rgw (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:18.771 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-07-17T12:00:18.783 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-07-17T12:00:18.813 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-17T12:00:19.058 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-07-17T12:00:19.070 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-07-17T12:00:19.099 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-07-17T12:00:19.427 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package liboath0. 2020-07-17T12:00:19.438 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-07-17T12:00:19.469 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-07-17T12:00:19.763 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libradosstriper1. 2020-07-17T12:00:19.776 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:19.813 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libradosstriper1 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:20.062 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-base. 2020-07-17T12:00:20.073 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../020-ceph-base_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:20.121 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-base (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:20.208 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-common. 2020-07-17T12:00:20.220 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../019-ceph-common_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:20.250 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-common (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:20.984 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-dateutil. 2020-07-17T12:00:20.997 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-07-17T12:00:21.038 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-07-17T12:00:21.346 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-asn1crypto. 2020-07-17T12:00:21.359 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-07-17T12:00:21.396 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-07-17T12:00:21.666 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cffi-backend. 2020-07-17T12:00:21.684 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-07-17T12:00:21.716 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-07-17T12:00:22.010 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cryptography. 2020-07-17T12:00:22.022 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-07-17T12:00:22.060 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-17T12:00:22.430 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-openssl. 2020-07-17T12:00:22.442 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-07-17T12:00:22.455 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-base. 2020-07-17T12:00:22.467 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../020-ceph-base_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:22.480 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-07-17T12:00:22.514 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-base (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:22.741 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-07-17T12:00:22.752 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-modules-core (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:23.128 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-bcrypt. 2020-07-17T12:00:23.141 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-07-17T12:00:23.170 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-07-17T12:00:23.437 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-dateutil. 2020-07-17T12:00:23.449 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-07-17T12:00:23.464 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cherrypy3. 2020-07-17T12:00:23.476 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-07-17T12:00:23.487 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-07-17T12:00:23.514 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-07-17T12:00:23.790 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-asn1crypto. 2020-07-17T12:00:23.801 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-07-17T12:00:23.826 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jwt. 2020-07-17T12:00:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-07-17T12:00:23.840 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-07-17T12:00:23.876 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-07-17T12:00:24.142 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cffi-backend. 2020-07-17T12:00:24.147 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-07-17T12:00:24.184 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-07-17T12:00:24.187 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-logutils. 2020-07-17T12:00:24.199 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-07-17T12:00:24.237 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-07-17T12:00:24.454 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cryptography. 2020-07-17T12:00:24.467 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-07-17T12:00:24.504 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-17T12:00:24.573 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-markupsafe. 2020-07-17T12:00:24.586 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-07-17T12:00:24.623 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-07-17T12:00:24.841 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-openssl. 2020-07-17T12:00:24.852 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-07-17T12:00:24.892 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-07-17T12:00:24.892 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-mako. 2020-07-17T12:00:24.905 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-07-17T12:00:24.934 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-07-17T12:00:25.160 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-07-17T12:00:25.171 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:25.203 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplegeneric. 2020-07-17T12:00:25.210 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-modules-core (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:25.214 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-07-17T12:00:25.254 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-07-17T12:00:25.540 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-singledispatch. 2020-07-17T12:00:25.547 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-bcrypt. 2020-07-17T12:00:25.553 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-07-17T12:00:25.560 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-07-17T12:00:25.590 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-07-17T12:00:25.597 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-07-17T12:00:25.867 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webob. 2020-07-17T12:00:25.880 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-07-17T12:00:25.908 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cherrypy3. 2020-07-17T12:00:25.917 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-07-17T12:00:25.921 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-07-17T12:00:25.958 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-07-17T12:00:26.328 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-jwt. 2020-07-17T12:00:26.340 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-07-17T12:00:26.378 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-07-17T12:00:26.520 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-bs4. 2020-07-17T12:00:26.532 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-07-17T12:00:26.570 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-07-17T12:00:26.715 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-logutils. 2020-07-17T12:00:26.725 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-07-17T12:00:26.765 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-07-17T12:00:26.881 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-waitress. 2020-07-17T12:00:26.892 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-07-17T12:00:26.923 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-07-17T12:00:27.068 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-markupsafe. 2020-07-17T12:00:27.081 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-07-17T12:00:27.118 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-07-17T12:00:27.201 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tempita. 2020-07-17T12:00:27.214 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-07-17T12:00:27.242 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-07-17T12:00:27.421 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-mako. 2020-07-17T12:00:27.522 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-07-17T12:00:27.522 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-07-17T12:00:27.545 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-paste. 2020-07-17T12:00:27.558 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-07-17T12:00:27.595 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-17T12:00:27.790 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-simplegeneric. 2020-07-17T12:00:27.802 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-07-17T12:00:27.840 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-07-17T12:00:27.915 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-07-17T12:00:27.928 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-07-17T12:00:27.965 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-07-17T12:00:28.176 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-singledispatch. 2020-07-17T12:00:28.188 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-07-17T12:00:28.226 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-07-17T12:00:28.267 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastedeploy. 2020-07-17T12:00:28.280 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-07-17T12:00:28.317 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-07-17T12:00:28.512 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-webob. 2020-07-17T12:00:28.523 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-07-17T12:00:28.562 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-07-17T12:00:28.603 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webtest. 2020-07-17T12:00:28.614 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-07-17T12:00:28.647 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-07-17T12:00:28.890 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-bs4. 2020-07-17T12:00:28.902 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-07-17T12:00:28.931 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-07-17T12:00:28.973 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pecan. 2020-07-17T12:00:28.984 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-07-17T12:00:29.023 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-07-17T12:00:29.243 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-waitress. 2020-07-17T12:00:29.260 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-07-17T12:00:29.294 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-07-17T12:00:29.411 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-werkzeug. 2020-07-17T12:00:29.424 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-07-17T12:00:29.463 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-17T12:00:29.629 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-tempita. 2020-07-17T12:00:29.641 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-07-17T12:00:29.679 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-07-17T12:00:29.814 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr. 2020-07-17T12:00:29.825 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:29.873 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:29.974 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-paste. 2020-07-17T12:00:29.985 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-07-17T12:00:30.023 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-17T12:00:30.333 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mon. 2020-07-17T12:00:30.345 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:30.352 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-07-17T12:00:30.364 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-07-17T12:00:30.383 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mon (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:30.402 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-07-17T12:00:30.663 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pastedeploy. 2020-07-17T12:00:30.676 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-07-17T12:00:30.738 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-07-17T12:00:31.032 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-webtest. 2020-07-17T12:00:31.044 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-07-17T12:00:31.082 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-07-17T12:00:31.306 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-osd. 2020-07-17T12:00:31.314 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:31.353 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-osd (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:31.452 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pecan. 2020-07-17T12:00:31.463 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-07-17T12:00:31.502 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-07-17T12:00:31.840 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-werkzeug. 2020-07-17T12:00:31.852 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-07-17T12:00:31.890 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-17T12:00:32.968 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr. 2020-07-17T12:00:32.980 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:33.018 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:33.343 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph. 2020-07-17T12:00:33.354 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../048-ceph_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:33.393 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:33.555 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mon. 2020-07-17T12:00:33.567 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:33.596 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mon (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:33.712 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-fuse. 2020-07-17T12:00:33.724 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:33.762 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-fuse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:34.161 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mds. 2020-07-17T12:00:34.168 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:34.206 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mds (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:34.549 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-osd. 2020-07-17T12:00:34.562 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:34.599 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-osd (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:34.693 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package cephadm. 2020-07-17T12:00:34.706 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../051-cephadm_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:34.743 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking cephadm (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:35.045 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jinja2. 2020-07-17T12:00:35.058 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-07-17T12:00:35.095 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-17T12:00:35.399 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-07-17T12:00:35.409 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:35.448 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-cephadm (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:35.751 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-repoze.lru. 2020-07-17T12:00:35.764 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-07-17T12:00:35.801 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-07-17T12:00:36.104 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-routes. 2020-07-17T12:00:36.122 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-07-17T12:00:36.154 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-07-17T12:00:36.680 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph. 2020-07-17T12:00:36.693 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../048-ceph_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:36.730 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:37.083 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-fuse. 2020-07-17T12:00:37.095 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:37.107 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-07-17T12:00:37.119 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:37.132 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-fuse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:37.161 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-dashboard (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:37.553 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mds. 2020-07-17T12:00:37.564 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:37.603 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mds (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:37.978 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-07-17T12:00:37.984 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:38.022 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-diskprediction-cloud (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:38.089 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package cephadm. 2020-07-17T12:00:38.104 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../051-cephadm_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:38.139 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking cephadm (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:38.333 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-numpy. 2020-07-17T12:00:38.345 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-07-17T12:00:38.383 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-17T12:00:38.476 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-jinja2. 2020-07-17T12:00:38.487 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-07-17T12:00:38.526 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-17T12:00:38.887 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-07-17T12:00:38.897 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:38.937 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-cephadm (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:38.981 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-decorator. 2020-07-17T12:00:38.992 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2020-07-17T12:00:39.030 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-07-17T12:00:39.240 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-repoze.lru. 2020-07-17T12:00:39.253 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-07-17T12:00:39.290 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-07-17T12:00:39.350 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-scipy. 2020-07-17T12:00:39.363 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-07-17T12:00:39.400 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-07-17T12:00:39.576 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-routes. 2020-07-17T12:00:39.592 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-07-17T12:00:39.618 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-07-17T12:00:39.912 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-07-17T12:00:39.924 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:39.962 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-dashboard (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:40.574 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-07-17T12:00:40.596 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-07-17T12:00:40.632 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-07-17T12:00:40.836 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-07-17T12:00:40.849 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:40.911 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-diskprediction-cloud (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:41.078 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-joblib. 2020-07-17T12:00:41.090 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2020-07-17T12:00:41.128 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-joblib (0.11-1) ... 2020-07-17T12:00:41.207 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-numpy. 2020-07-17T12:00:41.217 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-07-17T12:00:41.256 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-17T12:00:41.497 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn. 2020-07-17T12:00:41.509 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2020-07-17T12:00:41.543 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-07-17T12:00:41.870 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-decorator. 2020-07-17T12:00:41.884 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2020-07-17T12:00:41.928 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-07-17T12:00:42.079 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-07-17T12:00:42.092 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:42.129 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:42.281 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-scipy. 2020-07-17T12:00:42.294 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-07-17T12:00:42.356 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-07-17T12:00:43.218 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-monotonic. 2020-07-17T12:00:43.228 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2020-07-17T12:00:43.265 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-07-17T12:00:43.481 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-07-17T12:00:43.494 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-07-17T12:00:43.531 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-07-17T12:00:43.568 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-fasteners. 2020-07-17T12:00:43.580 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2020-07-17T12:00:43.618 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-07-17T12:00:43.887 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-httplib2. 2020-07-17T12:00:43.898 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-07-17T12:00:43.937 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-17T12:00:44.026 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-joblib. 2020-07-17T12:00:44.041 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2020-07-17T12:00:44.076 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-joblib (0.11-1) ... 2020-07-17T12:00:44.257 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1. 2020-07-17T12:00:44.270 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2020-07-17T12:00:44.307 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-07-17T12:00:44.429 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-sklearn. 2020-07-17T12:00:44.442 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2020-07-17T12:00:44.479 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-07-17T12:00:44.640 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-07-17T12:00:44.652 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-07-17T12:00:44.690 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-17T12:00:45.028 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-07-17T12:00:45.033 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rsa. 2020-07-17T12:00:45.040 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:45.045 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2020-07-17T12:00:45.077 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:45.083 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-07-17T12:00:45.344 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-07-17T12:00:45.358 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-07-17T12:00:45.394 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-17T12:00:45.831 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-oauth2client. 2020-07-17T12:00:45.844 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2020-07-17T12:00:45.881 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-07-17T12:00:46.131 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-monotonic. 2020-07-17T12:00:46.142 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2020-07-17T12:00:46.180 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-07-17T12:00:46.225 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-setuptools. 2020-07-17T12:00:46.237 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2020-07-17T12:00:46.270 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-07-17T12:00:46.516 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-fasteners. 2020-07-17T12:00:46.529 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2020-07-17T12:00:46.566 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-07-17T12:00:46.633 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-websocket. 2020-07-17T12:00:46.645 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-07-17T12:00:46.683 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-07-17T12:00:46.869 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-httplib2. 2020-07-17T12:00:46.881 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-07-17T12:00:46.919 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-17T12:00:46.986 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-kubernetes. 2020-07-17T12:00:46.999 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-07-17T12:00:47.036 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-17T12:00:47.231 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyasn1. 2020-07-17T12:00:47.242 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2020-07-17T12:00:47.282 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-07-17T12:00:47.457 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-07-17T12:00:47.469 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:47.507 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:47.584 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-07-17T12:00:47.596 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-07-17T12:00:47.652 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-17T12:00:47.810 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-json-pointer. 2020-07-17T12:00:47.816 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2020-07-17T12:00:47.853 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-07-17T12:00:47.982 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rsa. 2020-07-17T12:00:47.995 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2020-07-17T12:00:48.032 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-07-17T12:00:48.224 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jsonpatch. 2020-07-17T12:00:48.238 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-07-17T12:00:48.275 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-17T12:00:48.343 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-07-17T12:00:48.357 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-07-17T12:00:48.412 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-17T12:00:48.559 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-07-17T12:00:48.573 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../079-ceph-mgr-rook_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:48.617 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-rook (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-oauth2client. 2020-07-17T12:00:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2020-07-17T12:00:48.895 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libonig4:amd64. 2020-07-17T12:00:48.903 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-07-17T12:00:48.906 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2020-07-17T12:00:49.014 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-07-17T12:00:49.247 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-setuptools. 2020-07-17T12:00:49.261 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2020-07-17T12:00:49.305 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-07-17T12:00:49.366 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjq1:amd64. 2020-07-17T12:00:49.378 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2020-07-17T12:00:49.416 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-07-17T12:00:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package jq. 2020-07-17T12:00:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2020-07-17T12:00:49.711 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-07-17T12:00:49.725 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-websocket. 2020-07-17T12:00:49.737 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-07-17T12:00:49.775 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-07-17T12:00:49.946 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package socat. 2020-07-17T12:00:49.957 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-07-17T12:00:49.988 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-07-17T12:00:50.088 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-kubernetes. 2020-07-17T12:00:50.102 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-07-17T12:00:50.138 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-17T12:00:50.366 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-07-17T12:00:50.379 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-07-17T12:00:50.416 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-17T12:00:50.584 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-07-17T12:00:50.596 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:50.634 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:50.661 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package xmlstarlet. 2020-07-17T12:00:50.675 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2020-07-17T12:00:50.719 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-07-17T12:00:50.953 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-json-pointer. 2020-07-17T12:00:50.966 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2020-07-17T12:00:51.019 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-07-17T12:00:51.057 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-test. 2020-07-17T12:00:51.069 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../086-ceph-test_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:51.107 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-test (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:51.357 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-jsonpatch. 2020-07-17T12:00:51.370 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-07-17T12:00:51.409 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-17T12:00:51.735 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-07-17T12:00:51.747 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../079-ceph-mgr-rook_16.0.0-3281-g4dfb1075abb-1bionic_all.deb ... 2020-07-17T12:00:51.785 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-rook (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:52.095 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libonig4:amd64. 2020-07-17T12:00:52.107 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2020-07-17T12:00:52.233 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-07-17T12:00:52.850 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libjq1:amd64. 2020-07-17T12:00:52.865 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2020-07-17T12:00:52.900 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-07-17T12:00:53.171 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package jq. 2020-07-17T12:00:53.183 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2020-07-17T12:00:53.220 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-07-17T12:00:53.472 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package socat. 2020-07-17T12:00:53.484 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-07-17T12:00:53.522 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-07-17T12:00:53.959 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-07-17T12:00:53.971 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-07-17T12:00:54.009 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-17T12:00:54.270 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package xmlstarlet. 2020-07-17T12:00:54.281 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2020-07-17T12:00:54.312 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-07-17T12:00:54.626 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs-dev. 2020-07-17T12:00:54.639 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../087-libcephfs-dev_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:54.650 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-test. 2020-07-17T12:00:54.663 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../086-ceph-test_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:54.676 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs-dev (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:54.700 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-test (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:54.979 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-attr. 2020-07-17T12:00:54.990 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-07-17T12:00:55.029 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-attr (17.4.0-2) ... 2020-07-17T12:00:55.318 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-funcsigs. 2020-07-17T12:00:55.329 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-07-17T12:00:55.376 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-07-17T12:00:55.677 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-pluggy. 2020-07-17T12:00:55.689 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-07-17T12:00:55.726 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-07-17T12:00:56.021 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-py. 2020-07-17T12:00:56.033 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-07-17T12:00:56.071 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-py (1.5.2-1) ... 2020-07-17T12:00:56.398 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-pytest. 2020-07-17T12:00:56.409 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-07-17T12:00:56.450 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-07-17T12:00:57.015 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-attr. 2020-07-17T12:00:57.029 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-07-17T12:00:57.076 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-07-17T12:00:57.408 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cycler. 2020-07-17T12:00:57.421 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-07-17T12:00:57.454 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-07-17T12:00:57.759 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webencodings. 2020-07-17T12:00:57.767 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-07-17T12:00:57.805 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-07-17T12:00:58.107 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-html5lib. 2020-07-17T12:00:58.120 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-07-17T12:00:58.159 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-07-17T12:00:58.453 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libcephfs-dev. 2020-07-17T12:00:58.465 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../087-libcephfs-dev_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:00:58.503 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libcephfs-dev (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:00:58.515 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-07-17T12:00:58.528 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-07-17T12:00:58.565 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-17T12:00:58.823 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-attr. 2020-07-17T12:00:58.835 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-07-17T12:00:58.872 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-attr (17.4.0-2) ... 2020-07-17T12:00:58.993 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyparsing. 2020-07-17T12:00:59.005 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-07-17T12:00:59.044 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-17T12:00:59.225 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-funcsigs. 2020-07-17T12:00:59.245 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-07-17T12:00:59.283 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-07-17T12:00:59.376 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tz. 2020-07-17T12:00:59.388 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-07-17T12:00:59.429 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tz (2018.3-2) ... 2020-07-17T12:00:59.536 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-pluggy. 2020-07-17T12:00:59.548 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-07-17T12:00:59.578 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-07-17T12:00:59.725 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-matplotlib. 2020-07-17T12:00:59.736 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-07-17T12:00:59.780 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-17T12:00:59.902 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-py. 2020-07-17T12:00:59.915 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-07-17T12:00:59.954 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-py (1.5.2-1) ... 2020-07-17T12:01:00.263 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-pytest. 2020-07-17T12:01:00.277 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-07-17T12:01:00.322 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-07-17T12:01:00.466 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-olefile. 2020-07-17T12:01:00.479 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-07-17T12:01:00.524 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-07-17T12:01:00.699 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-attr. 2020-07-17T12:01:00.712 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-07-17T12:01:00.741 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-07-17T12:01:00.852 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastescript. 2020-07-17T12:01:00.864 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-07-17T12:01:00.893 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-07-17T12:01:01.075 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cycler. 2020-07-17T12:01:01.089 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-07-17T12:01:01.176 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-07-17T12:01:01.262 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pil:amd64. 2020-07-17T12:01:01.275 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.2_amd64.deb ... 2020-07-17T12:01:01.311 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-07-17T12:01:01.481 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-webencodings. 2020-07-17T12:01:01.492 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-07-17T12:01:01.525 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-07-17T12:01:01.674 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pluggy. 2020-07-17T12:01:01.686 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-07-17T12:01:01.732 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-07-17T12:01:01.828 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-html5lib. 2020-07-17T12:01:01.840 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-07-17T12:01:01.880 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-07-17T12:01:02.043 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-py. 2020-07-17T12:01:02.062 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-07-17T12:01:02.093 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-py (1.5.2-1) ... 2020-07-17T12:01:02.225 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-07-17T12:01:02.236 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-07-17T12:01:02.275 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-17T12:01:02.429 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyinotify. 2020-07-17T12:01:02.441 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-07-17T12:01:02.472 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-07-17T12:01:02.701 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyparsing. 2020-07-17T12:01:02.714 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-07-17T12:01:02.744 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-17T12:01:02.774 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pytest. 2020-07-17T12:01:02.787 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-07-17T12:01:02.825 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-07-17T12:01:03.020 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-tz. 2020-07-17T12:01:03.034 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-07-17T12:01:03.080 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-tz (2018.3-2) ... 2020-07-17T12:01:03.168 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplejson. 2020-07-17T12:01:03.181 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-07-17T12:01:03.226 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-07-17T12:01:03.406 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-matplotlib. 2020-07-17T12:01:03.419 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-07-17T12:01:03.457 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-17T12:01:03.564 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tk:amd64. 2020-07-17T12:01:03.575 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-07-17T12:01:03.615 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-17T12:01:04.080 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package radosgw. 2020-07-17T12:01:04.091 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../110-radosgw_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:01:04.130 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking radosgw (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:04.136 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-olefile. 2020-07-17T12:01:04.148 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-07-17T12:01:04.177 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-07-17T12:01:04.511 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pastescript. 2020-07-17T12:01:04.522 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-07-17T12:01:04.554 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-07-17T12:01:04.916 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pil:amd64. 2020-07-17T12:01:04.928 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.2_amd64.deb ... 2020-07-17T12:01:04.965 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-07-17T12:01:05.094 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package rbd-fuse. 2020-07-17T12:01:05.108 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:01:05.154 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking rbd-fuse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:05.369 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pluggy. 2020-07-17T12:01:05.383 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-07-17T12:01:05.429 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-07-17T12:01:05.454 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-07-17T12:01:05.711 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-py. 2020-07-17T12:01:05.725 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-07-17T12:01:05.733 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-17T12:01:05.772 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-py (1.5.2-1) ... 2020-07-17T12:01:06.121 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-17T12:01:06.152 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyinotify. 2020-07-17T12:01:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-07-17T12:01:06.202 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-07-17T12:01:06.232 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-17T12:01:06.525 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pytest. 2020-07-17T12:01:06.540 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-07-17T12:01:06.572 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-07-17T12:01:06.821 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-07-17T12:01:06.926 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-simplejson. 2020-07-17T12:01:06.937 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-07-17T12:01:06.975 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-07-17T12:01:07.220 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-07-17T12:01:07.312 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-tk:amd64. 2020-07-17T12:01:07.324 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-07-17T12:01:07.366 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-17T12:01:07.508 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webencodings (0.5-2) ... 2020-07-17T12:01:07.794 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package radosgw. 2020-07-17T12:01:07.796 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-attr (17.4.0-2) ... 2020-07-17T12:01:07.801 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../110-radosgw_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:01:07.840 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking radosgw (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:07.994 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-07-17T12:01:08.133 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-py (1.5.2-1) ... 2020-07-17T12:01:08.459 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-07-17T12:01:08.824 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-07-17T12:01:08.842 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package rbd-fuse. 2020-07-17T12:01:08.854 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-3281-g4dfb1075abb-1bionic_amd64.deb ... 2020-07-17T12:01:08.894 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking rbd-fuse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:09.149 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-07-17T12:01:09.209 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-07-17T12:01:09.424 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-argparse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:09.542 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-17T12:01:09.721 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-07-17T12:01:09.947 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-17T12:01:09.998 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-07-17T12:01:10.078 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-17T12:01:10.118 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-17T12:01:10.718 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-07-17T12:01:11.062 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-07-17T12:01:11.107 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-07-17T12:01:11.367 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-07-17T12:01:11.402 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-webencodings (0.5-2) ... 2020-07-17T12:01:11.644 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-py (1.5.2-1) ... 2020-07-17T12:01:11.686 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-attr (17.4.0-2) ... 2020-07-17T12:01:11.885 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-07-17T12:01:11.886 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-07-17T12:01:12.022 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-py (1.5.2-1) ... 2020-07-17T12:01:12.308 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-07-17T12:01:12.348 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-07-17T12:01:12.400 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-07-17T12:01:12.675 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-07-17T12:01:12.693 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-07-17T12:01:12.803 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-07-17T12:01:12.983 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-07-17T12:01:13.008 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-07-17T12:01:13.297 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-17T12:01:13.411 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-ceph-argparse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:13.574 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-07-17T12:01:13.723 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-07-17T12:01:13.862 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-07-17T12:01:14.020 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-07-17T12:01:14.133 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-17T12:01:14.249 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-07-17T12:01:14.554 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-07-17T12:01:14.843 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up cephadm (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:15.092 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-07-17T12:01:15.408 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-07-17T12:01:15.610 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user cephadm....done 2020-07-17T12:01:15.729 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-py (1.5.2-1) ... 2020-07-17T12:01:15.781 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-07-17T12:01:15.938 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-07-17T12:01:16.074 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-17T12:01:16.208 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-07-17T12:01:16.311 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-07-17T12:01:16.401 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-07-17T12:01:16.560 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-07-17T12:01:16.666 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-07-17T12:01:16.780 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-07-17T12:01:16.878 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-07-17T12:01:16.977 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-07-17T12:01:17.005 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-07-17T12:01:17.274 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-17T12:01:17.325 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-common (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:17.608 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-07-17T12:01:17.644 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-monotonic (1.1-2) ... 2020-07-17T12:01:17.896 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-07-17T12:01:17.908 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-17T12:01:18.038 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-17T12:01:18.277 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-07-17T12:01:18.383 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-07-17T12:01:18.607 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-07-17T12:01:18.726 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-17T12:01:18.844 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-07-17T12:01:18.885 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-07-17T12:01:18.904 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up cephadm (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:19.115 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-07-17T12:01:19.411 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-17T12:01:19.572 INFO:teuthology.orchestra.run.smithi052.stdout:Adding system user cephadm....done 2020-07-17T12:01:19.712 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-17T12:01:19.752 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-07-17T12:01:20.050 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-attr (17.4.0-2) ... 2020-07-17T12:01:20.053 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-17T12:01:20.181 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-07-17T12:01:20.351 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-07-17T12:01:20.519 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-07-17T12:01:20.555 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-17T12:01:20.775 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-07-17T12:01:20.894 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-07-17T12:01:20.952 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-07-17T12:01:21.050 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-07-17T12:01:21.241 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-ceph-common (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:21.290 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liboath0 (2.6.1-1) ... 2020-07-17T12:01:21.449 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-17T12:01:21.538 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-monotonic (1.1-2) ... 2020-07-17T12:01:21.579 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-17T12:01:21.800 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-17T12:01:21.822 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-07-17T12:01:21.926 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-17T12:01:21.968 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-07-17T12:01:22.297 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-07-17T12:01:22.364 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-17T12:01:22.465 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-07-17T12:01:22.632 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-17T12:01:22.730 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-07-17T12:01:22.753 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tz (2018.3-2) ... 2020-07-17T12:01:22.774 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-07-17T12:01:23.012 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-07-17T12:01:23.054 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-07-17T12:01:23.351 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-07-17T12:01:23.549 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-17T12:01:23.656 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-joblib (0.11-1) ... 2020-07-17T12:01:23.834 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-17T12:01:24.011 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-routes (2.4.1-1) ... 2020-07-17T12:01:24.192 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-attr (17.4.0-2) ... 2020-07-17T12:01:24.323 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-07-17T12:01:24.485 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-07-17T12:01:24.633 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-07-17T12:01:24.660 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-17T12:01:24.955 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-17T12:01:25.056 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-07-17T12:01:25.138 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-07-17T12:01:25.310 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-07-17T12:01:25.408 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up liboath0 (2.6.1-1) ... 2020-07-17T12:01:25.596 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-17T12:01:25.607 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up jq (1.5+dfsg-2) ... 2020-07-17T12:01:25.708 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-17T12:01:25.716 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-07-17T12:01:25.976 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-07-17T12:01:26.105 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-07-17T12:01:26.490 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-17T12:01:26.603 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-07-17T12:01:26.876 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-17T12:01:26.908 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-tz (2018.3-2) ... 2020-07-17T12:01:27.184 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-07-17T12:01:27.188 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-07-17T12:01:27.463 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-07-17T12:01:27.768 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-joblib (0.11-1) ... 2020-07-17T12:01:27.778 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-07-17T12:01:27.956 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-07-17T12:01:28.131 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-routes (2.4.1-1) ... 2020-07-17T12:01:28.334 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-pytest (3.3.2-2) ... 2020-07-17T12:01:28.416 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-07-17T12:01:28.594 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-07-17T12:01:28.699 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-07-17T12:01:28.703 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-07-17T12:01:28.816 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librados2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:28.937 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-07-17T12:01:29.025 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-17T12:01:29.250 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:29.376 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs-dev (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:29.397 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-07-17T12:01:29.497 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rados (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:29.615 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-07-17T12:01:29.673 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up jq (1.5+dfsg-2) ... 2020-07-17T12:01:29.794 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-07-17T12:01:29.927 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-07-17T12:01:30.217 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-07-17T12:01:30.568 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-17T12:01:30.993 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-17T12:01:31.321 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-07-17T12:01:31.420 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-07-17T12:01:31.644 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-07-17T12:01:31.770 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-07-17T12:01:32.133 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-pytest (3.3.2-2) ... 2020-07-17T12:01:32.348 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-07-17T12:01:32.379 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-07-17T12:01:32.489 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-07-17T12:01:32.593 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librados2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:32.636 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libradosstriper1 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:32.719 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-07-17T12:01:32.762 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cephfs (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:33.040 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libcephfs2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:33.060 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librgw2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:33.141 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libcephfs-dev (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:33.189 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-fuse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:33.270 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rados (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:33.394 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-07-17T12:01:33.397 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-17T12:01:33.398 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-17T12:01:33.692 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-07-17T12:01:33.694 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librbd1 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:33.815 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-07-17T12:01:33.896 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-07-17T12:01:33.926 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-07-17T12:01:33.990 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-07-17T12:01:34.224 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-17T12:01:34.315 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-17T12:01:34.885 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-modules-core (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:34.994 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-07-17T12:01:35.151 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-07-17T12:01:35.292 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rgw (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:35.401 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up rbd-fuse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:35.514 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rbd (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:35.637 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-common (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:36.136 INFO:teuthology.orchestra.run.smithi146.stdout:Adding group ceph....done 2020-07-17T12:01:36.145 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-07-17T12:01:36.433 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libradosstriper1 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:36.494 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user ceph....done 2020-07-17T12:01:36.534 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cephfs (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:36.655 INFO:teuthology.orchestra.run.smithi146.stdout:Setting system user ceph properties....done 2020-07-17T12:01:36.807 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-07-17T12:01:36.838 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librgw2 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:36.966 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-fuse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:37.024 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-07-17T12:01:37.188 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-17T12:01:37.189 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-17T12:01:37.294 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up radosgw (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:37.465 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librbd1 (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:37.525 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-17T12:01:37.525 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-17T12:01:37.569 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-07-17T12:01:37.684 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-07-17T12:01:37.725 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-07-17T12:01:37.978 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-test (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:38.019 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-17T12:01:38.104 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-base (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:38.433 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-07-17T12:01:38.705 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-modules-core (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:38.728 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:38.814 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-07-17T12:01:38.944 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-17T12:01:38.944 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-17T12:01:39.112 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rgw (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:39.213 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:39.238 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up rbd-fuse (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:39.355 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-osd (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:39.364 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rbd (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:39.465 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-common (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:39.792 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-17T12:01:39.792 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-17T12:01:39.965 INFO:teuthology.orchestra.run.smithi052.stdout:Adding group ceph....done 2020-07-17T12:01:40.061 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-cephadm (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:40.206 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-diskprediction-cloud (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:40.344 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-rook (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:40.363 INFO:teuthology.orchestra.run.smithi052.stdout:Adding system user ceph....done 2020-07-17T12:01:40.478 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mds (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:40.542 INFO:teuthology.orchestra.run.smithi052.stdout:Setting system user ceph properties....done 2020-07-17T12:01:40.697 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-07-17T12:01:40.701 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-17T12:01:40.701 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-17T12:01:40.935 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-07-17T12:01:40.963 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mon (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:41.177 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-17T12:01:41.178 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-17T12:01:41.220 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up radosgw (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:41.446 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-k8sevents (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:41.490 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-17T12:01:41.490 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-17T12:01:41.581 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-dashboard (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:41.715 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:41.840 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for systemd (237-3ubuntu10.41) ... 2020-07-17T12:01:41.880 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-test (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:42.006 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-base (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:42.024 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-07-17T12:01:42.315 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-07-17T12:01:42.605 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:42.818 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-17T12:01:42.819 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-17T12:01:43.074 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:43.200 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-osd (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:43.246 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-07-17T12:01:43.289 INFO:teuthology.orchestra.run.smithi146.stdout:ureadahead will be reprofiled on next reboot 2020-07-17T12:01:43.339 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-07-17T12:01:43.632 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-17T12:01:43.633 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-17T12:01:43.803 INFO:teuthology.orchestra.run.smithi146.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-07-17T12:01:43.806 DEBUG:teuthology.parallel:result is None 2020-07-17T12:01:43.973 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-cephadm (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:44.104 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-diskprediction-cloud (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:44.246 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-rook (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:44.414 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mds (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:44.633 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-17T12:01:44.634 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-17T12:01:44.922 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mon (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:45.163 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-17T12:01:45.164 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-17T12:01:45.432 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-k8sevents (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:45.533 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-dashboard (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:45.651 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph (16.0.0-3281-g4dfb1075abb-1bionic) ... 2020-07-17T12:01:45.784 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for systemd (237-3ubuntu10.41) ... 2020-07-17T12:01:45.985 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-07-17T12:01:47.272 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-07-17T12:01:47.315 INFO:teuthology.orchestra.run.smithi052.stdout:ureadahead will be reprofiled on next reboot 2020-07-17T12:01:47.373 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-07-17T12:01:47.823 INFO:teuthology.orchestra.run.smithi052.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-07-17T12:01:47.826 DEBUG:teuthology.parallel:result is None 2020-07-17T12:01:47.827 INFO:teuthology.orchestra.run.smithi146:> sudo lsb_release -is 2020-07-17T12:01:47.884 INFO:teuthology.orchestra.run.smithi146.stdout:Ubuntu 2020-07-17T12:01:47.890 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-17T12:01:47.892 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=4dfb1075abb69d7743ad054aace9aa373509ed36 2020-07-17T12:01:48.081 INFO:teuthology.orchestra.run.smithi146:> dpkg-query -W -f '${Version}' ceph 2020-07-17T12:01:48.106 INFO:teuthology.orchestra.run.smithi146.stdout:16.0.0-3281-g4dfb1075abb-1bionic 2020-07-17T12:01:48.107 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-3281-g4dfb1075abb-1bionic 2020-07-17T12:01:48.108 INFO:teuthology.task.install:The correct ceph version 16.0.0-3281-g4dfb1075abb-1bionic is installed. 2020-07-17T12:01:48.109 INFO:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2020-07-17T12:01:48.165 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2020-07-17T12:01:48.171 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-17T12:01:48.172 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=4dfb1075abb69d7743ad054aace9aa373509ed36 2020-07-17T12:01:48.401 INFO:teuthology.orchestra.run.smithi052:> dpkg-query -W -f '${Version}' ceph 2020-07-17T12:01:48.438 INFO:teuthology.orchestra.run.smithi052.stdout:16.0.0-3281-g4dfb1075abb-1bionic 2020-07-17T12:01:48.439 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-3281-g4dfb1075abb-1bionic 2020-07-17T12:01:48.439 INFO:teuthology.task.install:The correct ceph version 16.0.0-3281-g4dfb1075abb-1bionic is installed. 2020-07-17T12:01:48.440 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-07-17T12:01:48.440 INFO:teuthology.orchestra.run.smithi146:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-07-17T12:01:48.463 INFO:teuthology.orchestra.run.smithi052:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-07-17T12:01:48.508 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-07-17T12:01:48.509 INFO:teuthology.orchestra.run.smithi146:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-07-17T12:01:48.535 INFO:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-07-17T12:01:48.596 INFO:teuthology.orchestra.run.smithi052:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-07-17T12:01:48.613 INFO:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-07-17T12:01:48.671 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-07-17T12:01:48.671 INFO:teuthology.orchestra.run.smithi146:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-07-17T12:01:48.718 INFO:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-07-17T12:01:48.774 INFO:teuthology.orchestra.run.smithi052:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-07-17T12:01:48.822 INFO:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-07-17T12:01:48.885 INFO:teuthology.run_tasks:Running task ceph... 2020-07-17T12:01:50.079 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-07-17T12:01:50.080 INFO:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2020-07-17T12:01:50.083 INFO:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2020-07-17T12:01:50.096 INFO:tasks.ceph:Disabling ceph logrotate... 2020-07-17T12:01:50.097 INFO:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2020-07-17T12:01:50.143 INFO:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2020-07-17T12:01:50.164 INFO:tasks.ceph:Creating extra log directories... 2020-07-17T12:01:50.164 INFO:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-07-17T12:01:50.202 INFO:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-07-17T12:01:50.221 INFO:tasks.ceph:Creating ceph cluster c1... 2020-07-17T12:01:50.222 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-07-17T12:01:50.222 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi/5232112', 'branch': 'wip-dang-zipper-6', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5232112', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-dang-zipper-6'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-dang-zipper-6', 'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36'}}, 'owner': 'scheduled_dang@sidious.arb.redhat.com', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-dang-zipper-6', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-zipper-6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.8462', 'targets': {'smithi146.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCloEPXlv7laKq901S6qmHpGoaIMf1JrkGkSkzwRt3h1ROyX2Ct+bNpFeYY5wSXYHvAQpBj1UDv1akRuhp1ni69yiqm2fR5a8bOhfMPmVW4QXnLvOf1sD1gKiGBtXfdxqQqyM42035eYHaHay9laARJhsEQaNXuLUwL5AgSgUpxrzcN3vH6QryI0Cif3iCZwshjs2yG4pTBUwPcxrrh2W+xiV+YCJBwYsPMnUf6/9eZ0OFlbIva/mqMCjqF4EeJARYn4oqwwbdEuJBP9NGtNdvajktdr0wJoAXVRE+RDXtEiT7PHr5AzwNGq7ADl/s4HWZg4mUsSkuYzvcA+fHigPop', 'smithi052.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRym4vepUOl2/NXzCRLsZq2b/47oucdP+ocwQB0Bz64k/bJ0y8hUy3VGEfznebhWoaFcPdamYppBuqJ7dE+6H7xtNrKu+Vbp5WEGRFikdeCoLSb6iw0kbad+n82YKvd/K78sBc6uCstOykJumi82MlspRWXmUe873zCDXoGbQ/QAS/JCJ7a9FSyYL8zYkWZ2MueD01C0C3AGdrkecilm16odKiOnUhiU7Hi8+b5FY46K/FN4nKLYoTLXRWAlJRaOa9uc/lR1jwYb9UzewcyVNKLGLGG01TTvLkU7n04vqSfzxvrd5qzbtX6eseJjrxQeUGMsNwpxeHhRY+wSmkLBaD'}} 2020-07-17T12:01:50.223 INFO:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-07-17T12:01:50.258 INFO:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-07-17T12:01:50.272 INFO:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2020-07-17T12:01:50.309 INFO:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2020-07-17T12:01:50.379 DEBUG:teuthology.orchestra.remote:smithi146:/scratch_devs is 72B 2020-07-17T12:01:50.391 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-07-17T12:01:50.391 INFO:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-07-17T12:01:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-07-17T12:01:50.399 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-17T12:01:50.400 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-07-17T12:01:50.400 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-17T12:01:50.400 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2020-07-17 11:59:25.956068397 +0000 2020-07-17T12:01:50.400 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2020-07-17 11:59:25.844068398 +0000 2020-07-17T12:01:50.401 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2020-07-17 11:59:25.844068398 +0000 2020-07-17T12:01:50.401 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2020-07-17T12:01:50.408 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2020-07-17T12:01:50.409 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2020-07-17T12:01:50.409 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000351888 s, 1.5 MB/s 2020-07-17T12:01:50.413 INFO:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-07-17T12:01:50.462 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-07-17T12:01:50.462 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-17T12:01:50.462 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 347 Links: 1 2020-07-17T12:01:50.463 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-17T12:01:50.463 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2020-07-17 11:59:26.236068394 +0000 2020-07-17T12:01:50.463 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2020-07-17 11:59:26.104068395 +0000 2020-07-17T12:01:50.463 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2020-07-17 11:59:26.104068395 +0000 2020-07-17T12:01:50.463 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2020-07-17T12:01:50.470 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2020-07-17T12:01:50.471 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2020-07-17T12:01:50.471 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000357211 s, 1.4 MB/s 2020-07-17T12:01:50.475 INFO:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-07-17T12:01:50.523 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-07-17T12:01:50.524 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-17T12:01:50.524 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-07-17T12:01:50.524 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-17T12:01:50.525 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2020-07-17 11:59:26.392068392 +0000 2020-07-17T12:01:50.525 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2020-07-17 11:59:26.388068392 +0000 2020-07-17T12:01:50.525 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2020-07-17 11:59:26.388068392 +0000 2020-07-17T12:01:50.526 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2020-07-17T12:01:50.531 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2020-07-17T12:01:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2020-07-17T12:01:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000386013 s, 1.3 MB/s 2020-07-17T12:01:50.535 INFO:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-07-17T12:01:50.582 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-07-17T12:01:50.582 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-17T12:01:50.582 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-07-17T12:01:50.583 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-17T12:01:50.583 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2020-07-17 11:59:26.752068388 +0000 2020-07-17T12:01:50.583 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2020-07-17 11:59:26.648068389 +0000 2020-07-17T12:01:50.583 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2020-07-17 11:59:26.648068389 +0000 2020-07-17T12:01:50.583 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2020-07-17T12:01:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2020-07-17T12:01:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2020-07-17T12:01:50.592 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000422681 s, 1.2 MB/s 2020-07-17T12:01:50.595 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-17T12:01:50.596 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-17T12:01:50.596 INFO:tasks.ceph:Generating config... 2020-07-17T12:01:50.597 INFO:tasks.ceph:[client] debug rgw = 20 2020-07-17T12:01:50.598 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-07-17T12:01:50.598 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-07-17T12:01:50.599 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-17T12:01:50.599 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-07-17T12:01:50.599 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-07-17T12:01:50.600 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-07-17T12:01:50.600 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-07-17T12:01:50.600 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-07-17T12:01:50.600 INFO:tasks.ceph:[client] setgroup = ceph 2020-07-17T12:01:50.601 INFO:tasks.ceph:[client] setuser = ceph 2020-07-17T12:01:50.601 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-07-17T12:01:50.602 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-07-17T12:01:50.602 INFO:tasks.ceph:[mgr] debug ms = 1 2020-07-17T12:01:50.603 INFO:tasks.ceph:[mon] debug mon = 20 2020-07-17T12:01:50.603 INFO:tasks.ceph:[mon] debug ms = 1 2020-07-17T12:01:50.603 INFO:tasks.ceph:[mon] debug paxos = 20 2020-07-17T12:01:50.603 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-07-17T12:01:50.604 INFO:tasks.ceph:[osd] debug filestore = 20 2020-07-17T12:01:50.604 INFO:tasks.ceph:[osd] debug journal = 20 2020-07-17T12:01:50.604 INFO:tasks.ceph:[osd] debug ms = 20 2020-07-17T12:01:50.605 INFO:tasks.ceph:[osd] debug osd = 25 2020-07-17T12:01:50.605 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-07-17T12:01:50.605 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-07-17T12:01:50.606 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-07-17T12:01:50.606 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-07-17T12:01:50.606 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-07-17T12:01:50.607 INFO:tasks.ceph:Setting up c1.mon.a... 2020-07-17T12:01:50.607 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-07-17T12:01:50.667 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/c1.keyring 2020-07-17T12:01:50.669 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-07-17T12:01:50.745 INFO:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-07-17T12:01:50.800 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.146')] 2020-07-17T12:01:50.801 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.146 --print /home/ubuntu/cephtest/c1.monmap 2020-07-17T12:01:50.870 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-07-17T12:01:50.871 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: generated fsid db7266a8-b65a-4130-8823-0326cf7dfe98 2020-07-17T12:01:50.871 INFO:teuthology.orchestra.run.smithi146.stdout:epoch 0 2020-07-17T12:01:50.872 INFO:teuthology.orchestra.run.smithi146.stdout:fsid db7266a8-b65a-4130-8823-0326cf7dfe98 2020-07-17T12:01:50.872 INFO:teuthology.orchestra.run.smithi146.stdout:last_changed 2020-07-17T12:01:50.868312+0000 2020-07-17T12:01:50.872 INFO:teuthology.orchestra.run.smithi146.stdout:created 2020-07-17T12:01:50.868312+0000 2020-07-17T12:01:50.873 INFO:teuthology.orchestra.run.smithi146.stdout:min_mon_release 0 (unknown) 2020-07-17T12:01:50.873 INFO:teuthology.orchestra.run.smithi146.stdout:0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2020-07-17T12:01:50.873 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-07-17T12:01:50.875 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID db7266a8-b65a-4130-8823-0326cf7dfe98... 2020-07-17T12:01:50.877 INFO:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-07-17T12:01:50.881 INFO:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-07-17T12:01:50.918 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2020-07-17T12:01:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2020-07-17T12:01:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-17T12:01:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2020-07-17T12:01:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.919 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2020-07-17T12:01:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2020-07-17T12:01:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2020-07-17T12:01:50.920 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2020-07-17T12:01:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2020-07-17T12:01:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2020-07-17T12:01:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.921 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-17T12:01:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2020-07-17T12:01:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2020-07-17T12:01:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-17T12:01:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.922 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2020-07-17T12:01:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2020-07-17T12:01:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2020-07-17T12:01:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2020-07-17T12:01:50.923 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2020-07-17T12:01:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2020-07-17T12:01:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2020-07-17T12:01:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2020-07-17T12:01:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-17T12:01:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-17T12:01:50.924 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2020-07-17T12:01:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-17T12:01:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2020-07-17T12:01:50.925 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2020-07-17T12:01:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2020-07-17T12:01:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2020-07-17T12:01:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = 172.21.15.146 2020-07-17T12:01:50.926 INFO:teuthology.orchestra.run.smithi052.stdout: osd heartbeat grace = 40 2020-07-17T12:01:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = db7266a8-b65a-4130-8823-0326cf7dfe98 2020-07-17T12:01:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.927 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2020-07-17T12:01:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2020-07-17T12:01:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.928 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2020-07-17T12:01:50.928 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2020-07-17T12:01:50.928 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.928 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2020-07-17T12:01:50.928 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2020-07-17T12:01:50.928 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.929 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2020-07-17T12:01:50.929 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2020-07-17T12:01:50.929 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2020-07-17T12:01:50.929 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.929 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2020-07-17T12:01:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2020-07-17T12:01:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2020-07-17T12:01:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2020-07-17T12:01:50.930 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2020-07-17T12:01:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2020-07-17T12:01:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2020-07-17T12:01:50.931 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2020-07-17T12:01:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2020-07-17T12:01:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: debug filestore = 20 2020-07-17T12:01:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: debug journal = 20 2020-07-17T12:01:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 20 2020-07-17T12:01:50.932 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 25 2020-07-17T12:01:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: osd fast shutdown = False 2020-07-17T12:01:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: rocksdb delete range threshold = 0 2020-07-17T12:01:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.933 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2020-07-17T12:01:50.933 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2020-07-17T12:01:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2020-07-17T12:01:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2020-07-17T12:01:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2020-07-17T12:01:50.934 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2020-07-17T12:01:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2020-07-17T12:01:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2020-07-17T12:01:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.935 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2020-07-17T12:01:50.935 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2020-07-17T12:01:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2020-07-17T12:01:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2020-07-17T12:01:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2020-07-17T12:01:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2020-07-17T12:01:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2020-07-17T12:01:50.936 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2020-07-17T12:01:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2020-07-17T12:01:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2020-07-17T12:01:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2020-07-17T12:01:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd crush smoke test = False 2020-07-17T12:01:50.937 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:01:50.938 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2020-07-17T12:01:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2020-07-17T12:01:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2020-07-17T12:01:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2020-07-17T12:01:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-17T12:01:50.938 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-17T12:01:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw = 20 2020-07-17T12:01:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt require ssl = False 2020-07-17T12:01:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms backend = testing 2020-07-17T12:01:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-17T12:01:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: rgw curl low speed time = 300 2020-07-17T12:01:50.939 INFO:teuthology.orchestra.run.smithi052.stdout: rgw data log num shards = 4 2020-07-17T12:01:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: rgw md log max shards = 4 2020-07-17T12:01:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync log trim interval = 0 2020-07-17T12:01:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync obj etag verify = True 2020-07-17T12:01:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: setgroup = ceph 2020-07-17T12:01:50.940 INFO:teuthology.orchestra.run.smithi052.stdout: setuser = ceph 2020-07-17T12:01:50.941 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2020-07-17T12:01:50.941 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2020-07-17T12:01:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 10 2020-07-17T12:01:50.941 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2020-07-17T12:01:50.941 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 1000 2020-07-17T12:01:50.942 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2020-07-17T12:01:50.942 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 10000 2020-07-17T12:01:50.949 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2020-07-17T12:01:50.950 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2020-07-17T12:01:50.950 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-17T12:01:50.950 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2020-07-17T12:01:50.950 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.951 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2020-07-17T12:01:50.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.951 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2020-07-17T12:01:50.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.952 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2020-07-17T12:01:50.952 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2020-07-17T12:01:50.952 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.952 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2020-07-17T12:01:50.952 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2020-07-17T12:01:50.953 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.953 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-17T12:01:50.953 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2020-07-17T12:01:50.953 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.953 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2020-07-17T12:01:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-17T12:01:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2020-07-17T12:01:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2020-07-17T12:01:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2020-07-17T12:01:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2020-07-17T12:01:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2020-07-17T12:01:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2020-07-17T12:01:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2020-07-17T12:01:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2020-07-17T12:01:50.956 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-17T12:01:50.956 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-17T12:01:50.956 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2020-07-17T12:01:50.956 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.956 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-17T12:01:50.957 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.957 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2020-07-17T12:01:50.957 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.957 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2020-07-17T12:01:50.957 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.958 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2020-07-17T12:01:50.958 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2020-07-17T12:01:50.958 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = 172.21.15.146 2020-07-17T12:01:50.958 INFO:teuthology.orchestra.run.smithi146.stdout: osd heartbeat grace = 40 2020-07-17T12:01:50.958 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = db7266a8-b65a-4130-8823-0326cf7dfe98 2020-07-17T12:01:50.958 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.959 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2020-07-17T12:01:50.959 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2020-07-17T12:01:50.959 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.959 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2020-07-17T12:01:50.959 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2020-07-17T12:01:50.959 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.960 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2020-07-17T12:01:50.960 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2020-07-17T12:01:50.960 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.960 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2020-07-17T12:01:50.960 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2020-07-17T12:01:50.961 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2020-07-17T12:01:50.961 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.961 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2020-07-17T12:01:50.961 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2020-07-17T12:01:50.961 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.961 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2020-07-17T12:01:50.962 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.962 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2020-07-17T12:01:50.962 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.962 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2020-07-17T12:01:50.962 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2020-07-17T12:01:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2020-07-17T12:01:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2020-07-17T12:01:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2020-07-17T12:01:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: debug filestore = 20 2020-07-17T12:01:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: debug journal = 20 2020-07-17T12:01:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 20 2020-07-17T12:01:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 25 2020-07-17T12:01:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: osd fast shutdown = False 2020-07-17T12:01:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: rocksdb delete range threshold = 0 2020-07-17T12:01:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.965 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2020-07-17T12:01:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2020-07-17T12:01:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2020-07-17T12:01:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2020-07-17T12:01:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2020-07-17T12:01:50.966 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2020-07-17T12:01:50.966 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2020-07-17T12:01:50.966 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2020-07-17T12:01:50.966 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.966 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2020-07-17T12:01:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2020-07-17T12:01:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2020-07-17T12:01:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2020-07-17T12:01:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2020-07-17T12:01:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2020-07-17T12:01:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2020-07-17T12:01:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2020-07-17T12:01:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2020-07-17T12:01:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2020-07-17T12:01:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2020-07-17T12:01:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd crush smoke test = False 2020-07-17T12:01:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:01:50.969 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2020-07-17T12:01:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2020-07-17T12:01:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2020-07-17T12:01:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2020-07-17T12:01:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-17T12:01:50.970 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-17T12:01:50.970 INFO:teuthology.orchestra.run.smithi146.stdout: debug rgw = 20 2020-07-17T12:01:50.970 INFO:teuthology.orchestra.run.smithi146.stdout: rgw crypt require ssl = False 2020-07-17T12:01:50.970 INFO:teuthology.orchestra.run.smithi146.stdout: rgw crypt s3 kms backend = testing 2020-07-17T12:01:50.970 INFO:teuthology.orchestra.run.smithi146.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-17T12:01:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: rgw curl low speed time = 300 2020-07-17T12:01:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: rgw data log num shards = 4 2020-07-17T12:01:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: rgw md log max shards = 4 2020-07-17T12:01:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: rgw sync log trim interval = 0 2020-07-17T12:01:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: rgw sync obj etag verify = True 2020-07-17T12:01:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: setgroup = ceph 2020-07-17T12:01:50.972 INFO:teuthology.orchestra.run.smithi146.stdout: setuser = ceph 2020-07-17T12:01:50.972 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2020-07-17T12:01:50.972 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2020-07-17T12:01:50.972 INFO:teuthology.orchestra.run.smithi146.stdout: osd_max_omap_entries_per_request = 10 2020-07-17T12:01:50.972 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2020-07-17T12:01:50.972 INFO:teuthology.orchestra.run.smithi146.stdout: osd_max_omap_entries_per_request = 1000 2020-07-17T12:01:50.973 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2020-07-17T12:01:50.973 INFO:teuthology.orchestra.run.smithi146.stdout: osd_max_omap_entries_per_request = 10000 2020-07-17T12:01:50.974 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-07-17T12:01:50.975 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-07-17T12:01:51.060 INFO:tasks.ceph:Copying monmap to all nodes... 2020-07-17T12:01:51.112 DEBUG:teuthology.orchestra.remote:smithi146:/etc/ceph/c1.keyring is 206B 2020-07-17T12:01:51.215 DEBUG:teuthology.orchestra.remote:smithi146:/home/ubuntu/cephtest/c1.monmap is 179B 2020-07-17T12:01:51.267 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2020-07-17T12:01:51.268 INFO:teuthology.orchestra.run.smithi146:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-07-17T12:01:51.323 INFO:teuthology.orchestra.run.smithi146:> cat > /home/ubuntu/cephtest/c1.monmap 2020-07-17T12:01:51.409 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2020-07-17T12:01:51.409 INFO:teuthology.orchestra.run.smithi052:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-07-17T12:01:51.468 INFO:teuthology.orchestra.run.smithi052:> cat > /home/ubuntu/cephtest/c1.monmap 2020-07-17T12:01:51.557 INFO:tasks.ceph:Setting up mon nodes... 2020-07-17T12:01:51.558 INFO:tasks.ceph:Setting up mgr nodes... 2020-07-17T12:01:51.558 INFO:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-07-17T12:01:51.600 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-07-17T12:01:51.602 INFO:tasks.ceph:Setting up mds nodes... 2020-07-17T12:01:51.603 INFO:tasks.ceph_client:Setting up client nodes... 2020-07-17T12:01:51.603 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-07-17T12:01:51.683 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/c1.client.0.keyring 2020-07-17T12:01:51.694 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-07-17T12:01:51.771 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/c1.client.1.keyring 2020-07-17T12:01:51.783 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-07-17T12:01:51.784 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-17T12:01:51.785 INFO:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-07-17T12:01:51.846 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-17T12:01:51.847 INFO:tasks.ceph:role: c1.osd.0 2020-07-17T12:01:51.847 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2020-07-17T12:01:51.847 INFO:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-07-17T12:01:52.380 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-07-17T12:01:52.380 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-17T12:01:52.380 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-17T12:01:52.381 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-17T12:01:52.381 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2020-07-17T12:01:52.381 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-17T12:01:52.381 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-17T12:01:52.381 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-17T12:01:52.382 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-17T12:01:52.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2020-07-17T12:01:52.383 INFO:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-07-17T12:01:52.465 INFO:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-07-17T12:01:52.477 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: /sbin/restorecon: command not found 2020-07-17T12:01:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-17T12:01:52.480 INFO:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-07-17T12:01:52.534 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-17T12:01:52.535 INFO:tasks.ceph:role: c1.osd.1 2020-07-17T12:01:52.535 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2020-07-17T12:01:52.536 INFO:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-07-17T12:01:53.106 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-07-17T12:01:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-17T12:01:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-17T12:01:53.107 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-17T12:01:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2020-07-17T12:01:53.108 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-17T12:01:53.108 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-17T12:01:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-17T12:01:53.108 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-17T12:01:53.109 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2020-07-17T12:01:53.109 INFO:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-07-17T12:01:53.145 INFO:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-07-17T12:01:53.198 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: /sbin/restorecon: command not found 2020-07-17T12:01:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-17T12:01:53.200 INFO:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-07-17T12:01:53.257 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-17T12:01:53.257 INFO:tasks.ceph:role: c1.osd.2 2020-07-17T12:01:53.258 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2020-07-17T12:01:53.258 INFO:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-07-17T12:01:53.801 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-07-17T12:01:53.801 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-17T12:01:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-17T12:01:53.802 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-17T12:01:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2020-07-17T12:01:53.802 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-17T12:01:53.802 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-17T12:01:53.803 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-17T12:01:53.803 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-17T12:01:53.804 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2020-07-17T12:01:53.804 INFO:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-07-17T12:01:53.865 INFO:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-07-17T12:01:53.880 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: /sbin/restorecon: command not found 2020-07-17T12:01:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-17T12:01:53.881 INFO:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-07-17T12:01:53.977 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:53.972+0000 7fe9d1c75d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-07-17T12:01:53.977 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:53.972+0000 7fe9d1c75d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-07-17T12:01:53.978 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:53.972+0000 7fe9d1c75d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-07-17T12:01:53.978 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:53.972+0000 7fe9d1c75d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-07-17T12:01:53.978 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:53.972+0000 7fe9d1c75d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-07-17T12:01:53.980 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:53.976+0000 7fe9d1c75d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-07-17T12:01:55.899 INFO:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-07-17T12:01:55.915 INFO:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-07-17T12:01:55.993 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:55.988+0000 7faab14f6d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-07-17T12:01:55.994 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:55.988+0000 7faab14f6d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-07-17T12:01:55.994 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:55.988+0000 7faab14f6d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-07-17T12:01:55.995 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:55.988+0000 7faab14f6d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-07-17T12:01:55.995 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:55.988+0000 7faab14f6d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-07-17T12:01:55.996 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:55.988+0000 7faab14f6d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-07-17T12:01:57.911 INFO:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-07-17T12:01:57.968 INFO:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-07-17T12:01:58.049 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:58.044+0000 7fa6d0b42d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-07-17T12:01:58.050 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:58.044+0000 7fa6d0b42d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-07-17T12:01:58.050 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:58.044+0000 7fa6d0b42d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-07-17T12:01:58.050 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:58.044+0000 7fa6d0b42d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-07-17T12:01:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:58.044+0000 7fa6d0b42d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-07-17T12:01:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:01:58.044+0000 7fa6d0b42d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-07-17T12:01:59.986 INFO:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-07-17T12:02:00.042 INFO:tasks.ceph:Reading keys from all nodes... 2020-07-17T12:02:00.043 INFO:teuthology.orchestra.run.smithi146:> mktemp 2020-07-17T12:02:00.088 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.EqQGMTV1w4 2020-07-17T12:02:00.089 INFO:teuthology.orchestra.run.smithi146:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.EqQGMTV1w4 2020-07-17T12:02:00.145 INFO:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.EqQGMTV1w4 2020-07-17T12:02:00.251 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.EqQGMTV1w4 is 56B 2020-07-17T12:02:00.261 INFO:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.EqQGMTV1w4 2020-07-17T12:02:00.268 INFO:teuthology.orchestra.run.smithi146:> mktemp 2020-07-17T12:02:00.317 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.kVW8kSGjpS 2020-07-17T12:02:00.318 INFO:teuthology.orchestra.run.smithi146:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.kVW8kSGjpS 2020-07-17T12:02:00.373 INFO:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.kVW8kSGjpS 2020-07-17T12:02:00.483 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.kVW8kSGjpS is 56B 2020-07-17T12:02:00.536 INFO:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.kVW8kSGjpS 2020-07-17T12:02:00.542 INFO:teuthology.orchestra.run.smithi146:> mktemp 2020-07-17T12:02:00.593 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.ov8GbKejDa 2020-07-17T12:02:00.593 INFO:teuthology.orchestra.run.smithi146:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.ov8GbKejDa 2020-07-17T12:02:00.650 INFO:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.ov8GbKejDa 2020-07-17T12:02:00.755 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.ov8GbKejDa is 56B 2020-07-17T12:02:00.766 INFO:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.ov8GbKejDa 2020-07-17T12:02:00.772 INFO:teuthology.orchestra.run.smithi146:> mktemp 2020-07-17T12:02:00.821 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.etZMF7Jwb1 2020-07-17T12:02:00.823 INFO:teuthology.orchestra.run.smithi146:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.etZMF7Jwb1 2020-07-17T12:02:00.878 INFO:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.etZMF7Jwb1 2020-07-17T12:02:00.995 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.etZMF7Jwb1 is 56B 2020-07-17T12:02:01.053 INFO:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.etZMF7Jwb1 2020-07-17T12:02:01.111 DEBUG:teuthology.orchestra.remote:smithi146:/etc/ceph/c1.client.0.keyring is 59B 2020-07-17T12:02:01.130 DEBUG:teuthology.orchestra.remote:smithi146:/etc/ceph/c1.client.1.keyring is 59B 2020-07-17T12:02:01.140 INFO:tasks.ceph:Adding keys to all mons... 2020-07-17T12:02:01.141 INFO:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/c1.keyring 2020-07-17T12:02:01.188 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2020-07-17T12:02:01.188 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAvkxFf8/SkIxAAkUApfEyyrDsb1z8w8OUadg== 2020-07-17T12:02:01.189 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2020-07-17T12:02:01.189 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAxkxFfpP4XOhAAqLQkwDBJLKbuVYrX8Yuxug== 2020-07-17T12:02:01.196 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2020-07-17T12:02:01.198 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAzkxFf90MMOxAARmao5mVbBKt6kqJoIEPCHA== 2020-07-17T12:02:01.198 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2020-07-17T12:02:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQA2kxFfb0PHAhAAB0hoUGK0RsIYQQvyDe6f7A== 2020-07-17T12:02:01.199 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2020-07-17T12:02:01.199 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAvkxFfWZ+WKBAACaam7ysMiVRIBsQbXPN+Jw== 2020-07-17T12:02:01.199 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2020-07-17T12:02:01.200 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAvkxFfJFndLRAAw/RwyXuq3KHtzL1m4998Lg== 2020-07-17T12:02:01.201 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-07-17T12:02:01.285 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-17T12:02:01.372 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-17T12:02:01.460 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-17T12:02:01.545 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-17T12:02:01.632 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-17T12:02:01.671 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-07-17T12:02:01.671 INFO:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-07-17T12:02:01.726 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-07-17T12:02:01.826 INFO:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-07-17T12:02:01.840 INFO:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-07-17T12:02:01.889 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-07-17T12:02:01.890 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-07-17T12:02:01.891 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-07-17T12:02:01.933 INFO:tasks.ceph.c1.mon.a:Started 2020-07-17T12:02:01.934 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-07-17T12:02:01.934 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-07-17T12:02:01.935 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-07-17T12:02:01.938 INFO:tasks.ceph.c1.mgr.x:Started 2020-07-17T12:02:01.939 INFO:tasks.ceph:Setting crush tunables to default 2020-07-17T12:02:01.939 INFO:teuthology.orchestra.run.smithi146:> sudo ceph --cluster c1 osd crush tunables default 2020-07-17T12:02:02.326 INFO:teuthology.orchestra.run.smithi146.stderr:adjusted tunables profile to default 2020-07-17T12:02:02.341 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-07-17T12:02:02.342 INFO:teuthology.orchestra.run.smithi146:> mktemp 2020-07-17T12:02:02.348 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.PJeKUgIMPm 2020-07-17T12:02:02.348 INFO:teuthology.orchestra.run.smithi146:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.PJeKUgIMPm 2020-07-17T12:02:02.404 INFO:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.PJeKUgIMPm 2020-07-17T12:02:02.511 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.PJeKUgIMPm is 37B 2020-07-17T12:02:02.563 INFO:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.PJeKUgIMPm 2020-07-17T12:02:02.570 INFO:teuthology.orchestra.run.smithi146:> mktemp 2020-07-17T12:02:02.617 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.pltdjo3ZJX 2020-07-17T12:02:02.617 INFO:teuthology.orchestra.run.smithi146:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.pltdjo3ZJX 2020-07-17T12:02:02.674 INFO:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.pltdjo3ZJX 2020-07-17T12:02:02.779 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.pltdjo3ZJX is 37B 2020-07-17T12:02:02.794 INFO:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.pltdjo3ZJX 2020-07-17T12:02:02.812 INFO:teuthology.orchestra.run.smithi146:> mktemp 2020-07-17T12:02:02.861 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.hng8nwVooP 2020-07-17T12:02:02.861 INFO:teuthology.orchestra.run.smithi146:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.hng8nwVooP 2020-07-17T12:02:02.918 INFO:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.hng8nwVooP 2020-07-17T12:02:03.024 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.hng8nwVooP is 37B 2020-07-17T12:02:03.084 INFO:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.hng8nwVooP 2020-07-17T12:02:03.091 INFO:teuthology.orchestra.run.smithi146:> sudo ceph --cluster c1 osd new 7dc20e59-e20f-4bac-b42e-7e19ad527b62 0 2020-07-17T12:02:03.335 INFO:teuthology.orchestra.run.smithi146.stdout:0 2020-07-17T12:02:03.348 INFO:teuthology.orchestra.run.smithi146:> sudo ceph --cluster c1 osd new fe7f8306-7989-47a0-9562-bc11fb880f6e 1 2020-07-17T12:02:03.551 INFO:teuthology.orchestra.run.smithi146.stdout:1 2020-07-17T12:02:03.563 INFO:teuthology.orchestra.run.smithi146:> sudo ceph --cluster c1 osd new fee858a1-af9c-4299-befb-1907a82feabb 2 2020-07-17T12:02:03.760 INFO:teuthology.orchestra.run.smithi146.stdout:2 2020-07-17T12:02:03.772 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-07-17T12:02:03.772 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-07-17T12:02:03.774 INFO:tasks.ceph.c1.osd.0:Started 2020-07-17T12:02:03.775 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-07-17T12:02:03.775 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-07-17T12:02:03.777 INFO:tasks.ceph.c1.osd.1:Started 2020-07-17T12:02:03.777 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-07-17T12:02:03.778 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-07-17T12:02:03.779 INFO:tasks.ceph.c1.osd.2:Started 2020-07-17T12:02:03.780 INFO:tasks.ceph:Waiting for OSDs to come up 2020-07-17T12:02:04.083 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-07-17T12:02:04.153 INFO:tasks.ceph.c1.osd.1.smithi146.stderr:2020-07-17T12:02:04.148+0000 7f4c18fbad80 -1 Falling back to public interface 2020-07-17T12:02:04.252 INFO:teuthology.misc.health.smithi146.stdout: 2020-07-17T12:02:04.252 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":5,"fsid":"db7266a8-b65a-4130-8823-0326cf7dfe98","created":"2020-07-17T12:02:02.052739+0000","modified":"2020-07-17T12:02:03.755759+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7dc20e59-e20f-4bac-b42e-7e19ad527b62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fe7f8306-7989-47a0-9562-bc11fb880f6e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fee858a1-af9c-4299-befb-1907a82feabb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:04.269 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-07-17T12:02:04.389 INFO:tasks.ceph.c1.osd.2.smithi146.stderr:2020-07-17T12:02:04.384+0000 7f10bb594d80 -1 Falling back to public interface 2020-07-17T12:02:04.401 INFO:tasks.ceph.c1.osd.0.smithi146.stderr:2020-07-17T12:02:04.396+0000 7f412a15ad80 -1 Falling back to public interface 2020-07-17T12:02:04.781 INFO:tasks.ceph.c1.osd.1.smithi146.stderr:2020-07-17T12:02:04.776+0000 7f4c18fbad80 -1 osd.1 0 log_to_monitors {default=true} 2020-07-17T12:02:05.012 INFO:tasks.ceph.c1.osd.0.smithi146.stderr:2020-07-17T12:02:05.008+0000 7f412a15ad80 -1 osd.0 0 log_to_monitors {default=true} 2020-07-17T12:02:05.052 INFO:tasks.ceph.c1.osd.2.smithi146.stderr:2020-07-17T12:02:05.048+0000 7f10bb594d80 -1 osd.2 0 log_to_monitors {default=true} 2020-07-17T12:02:06.084 INFO:tasks.ceph.c1.osd.2.smithi146.stderr:2020-07-17T12:02:06.080+0000 7f10b2a5f700 -1 osd.2 0 waiting for initial osdmap 2020-07-17T12:02:06.084 INFO:tasks.ceph.c1.osd.0.smithi146.stderr:2020-07-17T12:02:06.080+0000 7f4121625700 -1 osd.0 0 waiting for initial osdmap 2020-07-17T12:02:06.085 INFO:tasks.ceph.c1.osd.1.smithi146.stderr:2020-07-17T12:02:06.080+0000 7f4c11c88700 -1 osd.1 0 waiting for initial osdmap 2020-07-17T12:02:10.573 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-07-17T12:02:10.829 INFO:teuthology.misc.health.smithi146.stdout: 2020-07-17T12:02:10.829 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":10,"fsid":"db7266a8-b65a-4130-8823-0326cf7dfe98","created":"2020-07-17T12:02:02.052739+0000","modified":"2020-07-17T12:02:09.098660+0000","last_up_change":"2020-07-17T12:02:07.075421+0000","last_in_change":"2020-07-17T12:02:07.075421+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-17T12:02:06.025512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"7dc20e59-e20f-4bac-b42e-7e19ad527b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":11402},{"type":"v1","addr":"172.21.15.146:6817","nonce":11402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":11402},{"type":"v1","addr":"172.21.15.146:6819","nonce":11402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":11402},{"type":"v1","addr":"172.21.15.146:6823","nonce":11402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":11402},{"type":"v1","addr":"172.21.15.146:6821","nonce":11402}]},"public_addr":"172.21.15.146:6817/11402","cluster_addr":"172.21.15.146:6819/11402","heartbeat_back_addr":"172.21.15.146:6823/11402","heartbeat_front_addr":"172.21.15.146:6821/11402","state":["exists","up"]},{"osd":1,"uuid":"fe7f8306-7989-47a0-9562-bc11fb880f6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":11400},{"type":"v1","addr":"172.21.15.146:6801","nonce":11400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":11400},{"type":"v1","addr":"172.21.15.146:6803","nonce":11400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":11400},{"type":"v1","addr":"172.21.15.146:6807","nonce":11400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":11400},{"type":"v1","addr":"172.21.15.146:6805","nonce":11400}]},"public_addr":"172.21.15.146:6801/11400","cluster_addr":"172.21.15.146:6803/11400","heartbeat_back_addr":"172.21.15.146:6807/11400","heartbeat_front_addr":"172.21.15.146:6805/11400","state":["exists","up"]},{"osd":2,"uuid":"fee858a1-af9c-4299-befb-1907a82feabb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":11401},{"type":"v1","addr":"172.21.15.146:6809","nonce":11401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":11401},{"type":"v1","addr":"172.21.15.146:6811","nonce":11401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":11401},{"type":"v1","addr":"172.21.15.146:6815","nonce":11401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":11401},{"type":"v1","addr":"172.21.15.146:6813","nonce":11401}]},"public_addr":"172.21.15.146:6809/11401","cluster_addr":"172.21.15.146:6811/11401","heartbeat_back_addr":"172.21.15.146:6815/11401","heartbeat_front_addr":"172.21.15.146:6813/11401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:06.008812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:05.761824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:06.068508+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:10.847 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-07-17T12:02:10.847 INFO:tasks.ceph:Creating RBD pool 2020-07-17T12:02:10.848 INFO:teuthology.orchestra.run.smithi146:> sudo ceph --cluster c1 osd pool create rbd 8 2020-07-17T12:02:12.134 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'rbd' created 2020-07-17T12:02:12.146 INFO:teuthology.orchestra.run.smithi146:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-07-17T12:02:13.140 INFO:teuthology.orchestra.run.smithi146.stderr:enabled application 'rbd' on pool 'rbd' 2020-07-17T12:02:13.151 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-07-17T12:02:13.151 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-07-17T12:02:13.152 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-07-17T12:02:13.419 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:13.420 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":12,"fsid":"db7266a8-b65a-4130-8823-0326cf7dfe98","created":"2020-07-17T12:02:02.052739+0000","modified":"2020-07-17T12:02:13.116651+0000","last_up_change":"2020-07-17T12:02:07.075421+0000","last_in_change":"2020-07-17T12:02:07.075421+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-17T12:02:06.025512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-17T12:02:11.109854+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7dc20e59-e20f-4bac-b42e-7e19ad527b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":11402},{"type":"v1","addr":"172.21.15.146:6817","nonce":11402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":11402},{"type":"v1","addr":"172.21.15.146:6819","nonce":11402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":11402},{"type":"v1","addr":"172.21.15.146:6823","nonce":11402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":11402},{"type":"v1","addr":"172.21.15.146:6821","nonce":11402}]},"public_addr":"172.21.15.146:6817/11402","cluster_addr":"172.21.15.146:6819/11402","heartbeat_back_addr":"172.21.15.146:6823/11402","heartbeat_front_addr":"172.21.15.146:6821/11402","state":["exists","up"]},{"osd":1,"uuid":"fe7f8306-7989-47a0-9562-bc11fb880f6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":11400},{"type":"v1","addr":"172.21.15.146:6801","nonce":11400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":11400},{"type":"v1","addr":"172.21.15.146:6803","nonce":11400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":11400},{"type":"v1","addr":"172.21.15.146:6807","nonce":11400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":11400},{"type":"v1","addr":"172.21.15.146:6805","nonce":11400}]},"public_addr":"172.21.15.146:6801/11400","cluster_addr":"172.21.15.146:6803/11400","heartbeat_back_addr":"172.21.15.146:6807/11400","heartbeat_front_addr":"172.21.15.146:6805/11400","state":["exists","up"]},{"osd":2,"uuid":"fee858a1-af9c-4299-befb-1907a82feabb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":11401},{"type":"v1","addr":"172.21.15.146:6809","nonce":11401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":11401},{"type":"v1","addr":"172.21.15.146:6811","nonce":11401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":11401},{"type":"v1","addr":"172.21.15.146:6815","nonce":11401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":11401},{"type":"v1","addr":"172.21.15.146:6813","nonce":11401}]},"public_addr":"172.21.15.146:6809/11401","cluster_addr":"172.21.15.146:6811/11401","heartbeat_back_addr":"172.21.15.146:6815/11401","heartbeat_front_addr":"172.21.15.146:6813/11401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:06.008812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:05.761824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:06.068508+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:13.433 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-07-17T12:02:06.025512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-07-17T12:02:11.109854+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-07-17T12:02:13.433 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2020-07-17T12:02:13.698 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2020-07-17T12:02:13.716 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2020-07-17T12:02:13.986 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 8 2020-07-17T12:02:13.999 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-07-17T12:02:14.000 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-07-17T12:02:14.000 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2020-07-17T12:02:14.292 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:14.412 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":11198},{"type":"v1","addr":"172.21.15.146:6825","nonce":11198}]},"active_addr":"172.21.15.146:6825/11198","active_change":"2020-07-17T12:02:05.983698+0000","active_mgr_features":4540138292837744639,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":2591919790}]},{"addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3363707396}]},{"addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":610632091}]}]}} 2020-07-17T12:02:14.415 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-07-17T12:02:14.415 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-07-17T12:02:14.416 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-07-17T12:02:14.689 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:14.689 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":13,"fsid":"db7266a8-b65a-4130-8823-0326cf7dfe98","created":"2020-07-17T12:02:02.052739+0000","modified":"2020-07-17T12:02:14.124786+0000","last_up_change":"2020-07-17T12:02:07.075421+0000","last_in_change":"2020-07-17T12:02:07.075421+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-17T12:02:06.025512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-17T12:02:11.109854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7dc20e59-e20f-4bac-b42e-7e19ad527b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":11402},{"type":"v1","addr":"172.21.15.146:6817","nonce":11402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":11402},{"type":"v1","addr":"172.21.15.146:6819","nonce":11402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":11402},{"type":"v1","addr":"172.21.15.146:6823","nonce":11402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":11402},{"type":"v1","addr":"172.21.15.146:6821","nonce":11402}]},"public_addr":"172.21.15.146:6817/11402","cluster_addr":"172.21.15.146:6819/11402","heartbeat_back_addr":"172.21.15.146:6823/11402","heartbeat_front_addr":"172.21.15.146:6821/11402","state":["exists","up"]},{"osd":1,"uuid":"fe7f8306-7989-47a0-9562-bc11fb880f6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":11400},{"type":"v1","addr":"172.21.15.146:6801","nonce":11400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":11400},{"type":"v1","addr":"172.21.15.146:6803","nonce":11400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":11400},{"type":"v1","addr":"172.21.15.146:6807","nonce":11400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":11400},{"type":"v1","addr":"172.21.15.146:6805","nonce":11400}]},"public_addr":"172.21.15.146:6801/11400","cluster_addr":"172.21.15.146:6803/11400","heartbeat_back_addr":"172.21.15.146:6807/11400","heartbeat_front_addr":"172.21.15.146:6805/11400","state":["exists","up"]},{"osd":2,"uuid":"fee858a1-af9c-4299-befb-1907a82feabb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":11401},{"type":"v1","addr":"172.21.15.146:6809","nonce":11401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":11401},{"type":"v1","addr":"172.21.15.146:6811","nonce":11401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":11401},{"type":"v1","addr":"172.21.15.146:6815","nonce":11401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":11401},{"type":"v1","addr":"172.21.15.146:6813","nonce":11401}]},"public_addr":"172.21.15.146:6809/11401","cluster_addr":"172.21.15.146:6811/11401","heartbeat_back_addr":"172.21.15.146:6815/11401","heartbeat_front_addr":"172.21.15.146:6813/11401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:06.008812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:05.761824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:06.068508+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:14.703 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-07-17T12:02:14.703 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-07-17T12:02:14.962 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:14.963 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":13,"fsid":"db7266a8-b65a-4130-8823-0326cf7dfe98","created":"2020-07-17T12:02:02.052739+0000","modified":"2020-07-17T12:02:14.124786+0000","last_up_change":"2020-07-17T12:02:07.075421+0000","last_in_change":"2020-07-17T12:02:07.075421+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-17T12:02:06.025512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-17T12:02:11.109854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7dc20e59-e20f-4bac-b42e-7e19ad527b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":11402},{"type":"v1","addr":"172.21.15.146:6817","nonce":11402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":11402},{"type":"v1","addr":"172.21.15.146:6819","nonce":11402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":11402},{"type":"v1","addr":"172.21.15.146:6823","nonce":11402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":11402},{"type":"v1","addr":"172.21.15.146:6821","nonce":11402}]},"public_addr":"172.21.15.146:6817/11402","cluster_addr":"172.21.15.146:6819/11402","heartbeat_back_addr":"172.21.15.146:6823/11402","heartbeat_front_addr":"172.21.15.146:6821/11402","state":["exists","up"]},{"osd":1,"uuid":"fe7f8306-7989-47a0-9562-bc11fb880f6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":11400},{"type":"v1","addr":"172.21.15.146:6801","nonce":11400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":11400},{"type":"v1","addr":"172.21.15.146:6803","nonce":11400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":11400},{"type":"v1","addr":"172.21.15.146:6807","nonce":11400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":11400},{"type":"v1","addr":"172.21.15.146:6805","nonce":11400}]},"public_addr":"172.21.15.146:6801/11400","cluster_addr":"172.21.15.146:6803/11400","heartbeat_back_addr":"172.21.15.146:6807/11400","heartbeat_front_addr":"172.21.15.146:6805/11400","state":["exists","up"]},{"osd":2,"uuid":"fee858a1-af9c-4299-befb-1907a82feabb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":11401},{"type":"v1","addr":"172.21.15.146:6809","nonce":11401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":11401},{"type":"v1","addr":"172.21.15.146:6811","nonce":11401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":11401},{"type":"v1","addr":"172.21.15.146:6815","nonce":11401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":11401},{"type":"v1","addr":"172.21.15.146:6813","nonce":11401}]},"public_addr":"172.21.15.146:6809/11401","cluster_addr":"172.21.15.146:6811/11401","heartbeat_back_addr":"172.21.15.146:6815/11401","heartbeat_front_addr":"172.21.15.146:6813/11401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:06.008812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:05.761824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:06.068508+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:14.968 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2020-07-17T12:02:15.129 INFO:teuthology.orchestra.run.smithi146.stdout:34359738371 2020-07-17T12:02:15.130 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2020-07-17T12:02:15.289 INFO:teuthology.orchestra.run.smithi146.stdout:34359738372 2020-07-17T12:02:15.290 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2020-07-17T12:02:15.450 INFO:teuthology.orchestra.run.smithi146.stdout:34359738372 2020-07-17T12:02:15.450 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-07-17T12:02:15.719 INFO:teuthology.orchestra.run.smithi146.stdout:34359738370 2020-07-17T12:02:15.738 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-07-17T12:02:16.738 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-07-17T12:02:17.012 INFO:teuthology.orchestra.run.smithi146.stdout:34359738371 2020-07-17T12:02:17.025 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2020-07-17T12:02:17.026 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2020-07-17T12:02:17.290 INFO:teuthology.orchestra.run.smithi146.stdout:34359738372 2020-07-17T12:02:17.303 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2020-07-17T12:02:17.303 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2020-07-17T12:02:17.571 INFO:teuthology.orchestra.run.smithi146.stdout:34359738372 2020-07-17T12:02:17.582 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2020-07-17T12:02:17.583 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-07-17T12:02:17.583 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-07-17T12:02:17.841 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:17.843 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2020-07-17T12:02:17.854 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-07-17T12:02:15.994446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.874960"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.134755+0000","last_change":"2020-07-17T12:02:13.133464+0000","last_active":"2020-07-17T12:02:13.134755+0000","last_peered":"2020-07-17T12:02:13.134755+0000","last_clean":"2020-07-17T12:02:13.134755+0000","last_became_active":"2020-07-17T12:02:13.131751+0000","last_became_peered":"2020-07-17T12:02:13.131751+0000","last_unstale":"2020-07-17T12:02:13.134755+0000","last_undegraded":"2020-07-17T12:02:13.134755+0000","last_fullsized":"2020-07-17T12:02:13.134755+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.133957+0000","last_change":"2020-07-17T12:02:13.132627+0000","last_active":"2020-07-17T12:02:13.133957+0000","last_peered":"2020-07-17T12:02:13.133957+0000","last_clean":"2020-07-17T12:02:13.133957+0000","last_became_active":"2020-07-17T12:02:13.132188+0000","last_became_peered":"2020-07-17T12:02:13.132188+0000","last_unstale":"2020-07-17T12:02:13.133957+0000","last_undegraded":"2020-07-17T12:02:13.133957+0000","last_fullsized":"2020-07-17T12:02:13.133957+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.135895+0000","last_change":"2020-07-17T12:02:13.134557+0000","last_active":"2020-07-17T12:02:13.135895+0000","last_peered":"2020-07-17T12:02:13.135895+0000","last_clean":"2020-07-17T12:02:13.135895+0000","last_became_active":"2020-07-17T12:02:13.134085+0000","last_became_peered":"2020-07-17T12:02:13.134085+0000","last_unstale":"2020-07-17T12:02:13.135895+0000","last_undegraded":"2020-07-17T12:02:13.135895+0000","last_fullsized":"2020-07-17T12:02:13.135895+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.132738+0000","last_change":"2020-07-17T12:02:13.132738+0000","last_active":"2020-07-17T12:02:13.132738+0000","last_peered":"2020-07-17T12:02:13.132738+0000","last_clean":"2020-07-17T12:02:13.132738+0000","last_became_active":"2020-07-17T12:02:13.132327+0000","last_became_peered":"2020-07-17T12:02:13.132327+0000","last_unstale":"2020-07-17T12:02:13.132738+0000","last_undegraded":"2020-07-17T12:02:13.132738+0000","last_fullsized":"2020-07-17T12:02:13.132738+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.130361+0000","last_change":"2020-07-17T12:02:08.982759+0000","last_active":"2020-07-17T12:02:13.130361+0000","last_peered":"2020-07-17T12:02:13.130361+0000","last_clean":"2020-07-17T12:02:13.130361+0000","last_became_active":"2020-07-17T12:02:08.981119+0000","last_became_peered":"2020-07-17T12:02:08.981119+0000","last_unstale":"2020-07-17T12:02:13.130361+0000","last_undegraded":"2020-07-17T12:02:13.130361+0000","last_fullsized":"2020-07-17T12:02:13.130361+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:06.068048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:06.068048+0000","last_clean_scrub_stamp":"2020-07-17T12:02:06.068048+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.134809+0000","last_change":"2020-07-17T12:02:13.133606+0000","last_active":"2020-07-17T12:02:13.134809+0000","last_peered":"2020-07-17T12:02:13.134809+0000","last_clean":"2020-07-17T12:02:13.134809+0000","last_became_active":"2020-07-17T12:02:13.132890+0000","last_became_peered":"2020-07-17T12:02:13.132890+0000","last_unstale":"2020-07-17T12:02:13.134809+0000","last_undegraded":"2020-07-17T12:02:13.134809+0000","last_fullsized":"2020-07-17T12:02:13.134809+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.133089+0000","last_change":"2020-07-17T12:02:13.133089+0000","last_active":"2020-07-17T12:02:13.133089+0000","last_peered":"2020-07-17T12:02:13.133089+0000","last_clean":"2020-07-17T12:02:13.133089+0000","last_became_active":"2020-07-17T12:02:13.131848+0000","last_became_peered":"2020-07-17T12:02:13.131848+0000","last_unstale":"2020-07-17T12:02:13.133089+0000","last_undegraded":"2020-07-17T12:02:13.133089+0000","last_fullsized":"2020-07-17T12:02:13.133089+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.136555+0000","last_change":"2020-07-17T12:02:13.135944+0000","last_active":"2020-07-17T12:02:13.136555+0000","last_peered":"2020-07-17T12:02:13.136555+0000","last_clean":"2020-07-17T12:02:13.136555+0000","last_became_active":"2020-07-17T12:02:13.135484+0000","last_became_peered":"2020-07-17T12:02:13.135484+0000","last_unstale":"2020-07-17T12:02:13.136555+0000","last_undegraded":"2020-07-17T12:02:13.136555+0000","last_fullsized":"2020-07-17T12:02:13.136555+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.134237+0000","last_change":"2020-07-17T12:02:13.133301+0000","last_active":"2020-07-17T12:02:13.134237+0000","last_peered":"2020-07-17T12:02:13.134237+0000","last_clean":"2020-07-17T12:02:13.134237+0000","last_became_active":"2020-07-17T12:02:13.132865+0000","last_became_peered":"2020-07-17T12:02:13.132865+0000","last_unstale":"2020-07-17T12:02:13.134237+0000","last_undegraded":"2020-07-17T12:02:13.134237+0000","last_fullsized":"2020-07-17T12:02:13.134237+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7889999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5589999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0539999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]}]}]}} 2020-07-17T12:02:17.855 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-07-17T12:02:18.116 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2020-07-17T12:02:18.129 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-07-17T12:02:17.994985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.875499"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.134755+0000","last_change":"2020-07-17T12:02:13.133464+0000","last_active":"2020-07-17T12:02:13.134755+0000","last_peered":"2020-07-17T12:02:13.134755+0000","last_clean":"2020-07-17T12:02:13.134755+0000","last_became_active":"2020-07-17T12:02:13.131751+0000","last_became_peered":"2020-07-17T12:02:13.131751+0000","last_unstale":"2020-07-17T12:02:13.134755+0000","last_undegraded":"2020-07-17T12:02:13.134755+0000","last_fullsized":"2020-07-17T12:02:13.134755+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.133957+0000","last_change":"2020-07-17T12:02:13.132627+0000","last_active":"2020-07-17T12:02:13.133957+0000","last_peered":"2020-07-17T12:02:13.133957+0000","last_clean":"2020-07-17T12:02:13.133957+0000","last_became_active":"2020-07-17T12:02:13.132188+0000","last_became_peered":"2020-07-17T12:02:13.132188+0000","last_unstale":"2020-07-17T12:02:13.133957+0000","last_undegraded":"2020-07-17T12:02:13.133957+0000","last_fullsized":"2020-07-17T12:02:13.133957+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.135895+0000","last_change":"2020-07-17T12:02:13.134557+0000","last_active":"2020-07-17T12:02:13.135895+0000","last_peered":"2020-07-17T12:02:13.135895+0000","last_clean":"2020-07-17T12:02:13.135895+0000","last_became_active":"2020-07-17T12:02:13.134085+0000","last_became_peered":"2020-07-17T12:02:13.134085+0000","last_unstale":"2020-07-17T12:02:13.135895+0000","last_undegraded":"2020-07-17T12:02:13.135895+0000","last_fullsized":"2020-07-17T12:02:13.135895+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.132738+0000","last_change":"2020-07-17T12:02:13.132738+0000","last_active":"2020-07-17T12:02:13.132738+0000","last_peered":"2020-07-17T12:02:13.132738+0000","last_clean":"2020-07-17T12:02:13.132738+0000","last_became_active":"2020-07-17T12:02:13.132327+0000","last_became_peered":"2020-07-17T12:02:13.132327+0000","last_unstale":"2020-07-17T12:02:13.132738+0000","last_undegraded":"2020-07-17T12:02:13.132738+0000","last_fullsized":"2020-07-17T12:02:13.132738+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.130361+0000","last_change":"2020-07-17T12:02:08.982759+0000","last_active":"2020-07-17T12:02:13.130361+0000","last_peered":"2020-07-17T12:02:13.130361+0000","last_clean":"2020-07-17T12:02:13.130361+0000","last_became_active":"2020-07-17T12:02:08.981119+0000","last_became_peered":"2020-07-17T12:02:08.981119+0000","last_unstale":"2020-07-17T12:02:13.130361+0000","last_undegraded":"2020-07-17T12:02:13.130361+0000","last_fullsized":"2020-07-17T12:02:13.130361+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:06.068048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:06.068048+0000","last_clean_scrub_stamp":"2020-07-17T12:02:06.068048+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.134809+0000","last_change":"2020-07-17T12:02:13.133606+0000","last_active":"2020-07-17T12:02:13.134809+0000","last_peered":"2020-07-17T12:02:13.134809+0000","last_clean":"2020-07-17T12:02:13.134809+0000","last_became_active":"2020-07-17T12:02:13.132890+0000","last_became_peered":"2020-07-17T12:02:13.132890+0000","last_unstale":"2020-07-17T12:02:13.134809+0000","last_undegraded":"2020-07-17T12:02:13.134809+0000","last_fullsized":"2020-07-17T12:02:13.134809+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.133089+0000","last_change":"2020-07-17T12:02:13.133089+0000","last_active":"2020-07-17T12:02:13.133089+0000","last_peered":"2020-07-17T12:02:13.133089+0000","last_clean":"2020-07-17T12:02:13.133089+0000","last_became_active":"2020-07-17T12:02:13.131848+0000","last_became_peered":"2020-07-17T12:02:13.131848+0000","last_unstale":"2020-07-17T12:02:13.133089+0000","last_undegraded":"2020-07-17T12:02:13.133089+0000","last_fullsized":"2020-07-17T12:02:13.133089+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.136555+0000","last_change":"2020-07-17T12:02:13.135944+0000","last_active":"2020-07-17T12:02:13.136555+0000","last_peered":"2020-07-17T12:02:13.136555+0000","last_clean":"2020-07-17T12:02:13.136555+0000","last_became_active":"2020-07-17T12:02:13.135484+0000","last_became_peered":"2020-07-17T12:02:13.135484+0000","last_unstale":"2020-07-17T12:02:13.136555+0000","last_undegraded":"2020-07-17T12:02:13.136555+0000","last_fullsized":"2020-07-17T12:02:13.136555+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:13.134237+0000","last_change":"2020-07-17T12:02:13.133301+0000","last_active":"2020-07-17T12:02:13.134237+0000","last_peered":"2020-07-17T12:02:13.134237+0000","last_clean":"2020-07-17T12:02:13.134237+0000","last_became_active":"2020-07-17T12:02:13.132865+0000","last_became_peered":"2020-07-17T12:02:13.132865+0000","last_unstale":"2020-07-17T12:02:13.134237+0000","last_undegraded":"2020-07-17T12:02:13.134237+0000","last_fullsized":"2020-07-17T12:02:13.134237+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:12.106874+0000","last_clean_scrub_stamp":"2020-07-17T12:02:12.106874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7889999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5589999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0539999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]}]}]}} 2020-07-17T12:02:18.130 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-07-17T12:02:18.131 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-07-17T12:02:18.131 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-07-17T12:02:18.132 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2020-07-17T12:02:18.439 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:18.451 INFO:teuthology.orchestra.run.smithi146.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-07-17T12:02:18.452 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-07-17T12:02:18.453 INFO:teuthology.run_tasks:Running task ceph... 2020-07-17T12:02:18.463 INFO:tasks.ceph:Creating ceph cluster c2... 2020-07-17T12:02:18.463 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-07-17T12:02:18.464 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi/5232112', 'branch': 'wip-dang-zipper-6', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5232112', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dang-2020-07-16_20:32:24-rgw-wip-dang-zipper-6-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-dang-zipper-6'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-dang-zipper-6', 'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36'}}, 'owner': 'scheduled_dang@sidious.arb.redhat.com', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-dang-zipper-6', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-zipper-6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '4dfb1075abb69d7743ad054aace9aa373509ed36', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.8462', 'targets': {'smithi146.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCloEPXlv7laKq901S6qmHpGoaIMf1JrkGkSkzwRt3h1ROyX2Ct+bNpFeYY5wSXYHvAQpBj1UDv1akRuhp1ni69yiqm2fR5a8bOhfMPmVW4QXnLvOf1sD1gKiGBtXfdxqQqyM42035eYHaHay9laARJhsEQaNXuLUwL5AgSgUpxrzcN3vH6QryI0Cif3iCZwshjs2yG4pTBUwPcxrrh2W+xiV+YCJBwYsPMnUf6/9eZ0OFlbIva/mqMCjqF4EeJARYn4oqwwbdEuJBP9NGtNdvajktdr0wJoAXVRE+RDXtEiT7PHr5AzwNGq7ADl/s4HWZg4mUsSkuYzvcA+fHigPop', 'smithi052.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRym4vepUOl2/NXzCRLsZq2b/47oucdP+ocwQB0Bz64k/bJ0y8hUy3VGEfznebhWoaFcPdamYppBuqJ7dE+6H7xtNrKu+Vbp5WEGRFikdeCoLSb6iw0kbad+n82YKvd/K78sBc6uCstOykJumi82MlspRWXmUe873zCDXoGbQ/QAS/JCJ7a9FSyYL8zYkWZ2MueD01C0C3AGdrkecilm16odKiOnUhiU7Hi8+b5FY46K/FN4nKLYoTLXRWAlJRaOa9uc/lR1jwYb9UzewcyVNKLGLGG01TTvLkU7n04vqSfzxvrd5qzbtX6eseJjrxQeUGMsNwpxeHhRY+wSmkLBaD'}} 2020-07-17T12:02:18.465 INFO:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-07-17T12:02:18.468 INFO:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-07-17T12:02:18.475 INFO:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2020-07-17T12:02:18.517 INFO:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2020-07-17T12:02:18.579 DEBUG:teuthology.orchestra.remote:smithi052:/scratch_devs is 72B 2020-07-17T12:02:18.590 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-07-17T12:02:18.591 INFO:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-07-17T12:02:18.598 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-07-17T12:02:18.598 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-17T12:02:18.598 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-07-17T12:02:18.599 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-17T12:02:18.599 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2020-07-17 11:59:25.691539147 +0000 2020-07-17T12:02:18.599 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2020-07-17 11:59:25.575481150 +0000 2020-07-17T12:02:18.599 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2020-07-17 11:59:25.575481150 +0000 2020-07-17T12:02:18.599 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2020-07-17T12:02:18.605 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2020-07-17T12:02:18.606 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2020-07-17T12:02:18.606 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000274829 s, 1.9 MB/s 2020-07-17T12:02:18.610 INFO:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-07-17T12:02:18.658 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-07-17T12:02:18.658 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-17T12:02:18.658 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-07-17T12:02:18.659 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-17T12:02:18.659 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2020-07-17 11:59:25.995691141 +0000 2020-07-17T12:02:18.659 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2020-07-17 11:59:25.867627144 +0000 2020-07-17T12:02:18.659 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2020-07-17 11:59:25.867627144 +0000 2020-07-17T12:02:18.659 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2020-07-17T12:02:18.667 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2020-07-17T12:02:18.667 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2020-07-17T12:02:18.667 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.00028726 s, 1.8 MB/s 2020-07-17T12:02:18.671 INFO:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-07-17T12:02:18.722 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-07-17T12:02:18.723 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-17T12:02:18.723 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-07-17T12:02:18.723 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-17T12:02:18.723 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2020-07-17 11:59:26.235811135 +0000 2020-07-17T12:02:18.724 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2020-07-17 11:59:26.139763137 +0000 2020-07-17T12:02:18.724 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2020-07-17 11:59:26.139763137 +0000 2020-07-17T12:02:18.724 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2020-07-17T12:02:18.731 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2020-07-17T12:02:18.732 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2020-07-17T12:02:18.732 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.00026001 s, 2.0 MB/s 2020-07-17T12:02:18.736 INFO:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-07-17T12:02:18.786 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-07-17T12:02:18.786 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-17T12:02:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-07-17T12:02:18.787 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-17T12:02:18.787 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2020-07-17 11:59:26.475931130 +0000 2020-07-17T12:02:18.787 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2020-07-17 11:59:26.347867133 +0000 2020-07-17T12:02:18.787 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2020-07-17 11:59:26.347867133 +0000 2020-07-17T12:02:18.787 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2020-07-17T12:02:18.796 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2020-07-17T12:02:18.796 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2020-07-17T12:02:18.796 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000268304 s, 1.9 MB/s 2020-07-17T12:02:18.800 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-17T12:02:18.801 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-17T12:02:18.802 INFO:tasks.ceph:Generating config... 2020-07-17T12:02:18.804 INFO:tasks.ceph:[client] debug rgw = 20 2020-07-17T12:02:18.804 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-07-17T12:02:18.805 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-07-17T12:02:18.805 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-17T12:02:18.805 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-07-17T12:02:18.806 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-07-17T12:02:18.806 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-07-17T12:02:18.806 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-07-17T12:02:18.807 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-07-17T12:02:18.807 INFO:tasks.ceph:[client] setgroup = ceph 2020-07-17T12:02:18.807 INFO:tasks.ceph:[client] setuser = ceph 2020-07-17T12:02:18.807 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-07-17T12:02:18.808 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-07-17T12:02:18.808 INFO:tasks.ceph:[mgr] debug ms = 1 2020-07-17T12:02:18.808 INFO:tasks.ceph:[mon] debug mon = 20 2020-07-17T12:02:18.808 INFO:tasks.ceph:[mon] debug ms = 1 2020-07-17T12:02:18.809 INFO:tasks.ceph:[mon] debug paxos = 20 2020-07-17T12:02:18.809 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-07-17T12:02:18.809 INFO:tasks.ceph:[osd] debug filestore = 20 2020-07-17T12:02:18.810 INFO:tasks.ceph:[osd] debug journal = 20 2020-07-17T12:02:18.810 INFO:tasks.ceph:[osd] debug ms = 20 2020-07-17T12:02:18.810 INFO:tasks.ceph:[osd] debug osd = 25 2020-07-17T12:02:18.811 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-07-17T12:02:18.811 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-07-17T12:02:18.811 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-07-17T12:02:18.811 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-07-17T12:02:18.812 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-07-17T12:02:18.813 INFO:tasks.ceph:Setting up c2.mon.a... 2020-07-17T12:02:18.813 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-07-17T12:02:18.875 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/c2.keyring 2020-07-17T12:02:18.877 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-07-17T12:02:18.953 INFO:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-07-17T12:02:19.012 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.52')] 2020-07-17T12:02:19.012 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.52 --print /home/ubuntu/cephtest/c2.monmap 2020-07-17T12:02:19.084 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-07-17T12:02:19.084 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: generated fsid 4481a3ce-4482-458d-bce8-7b9423f8702f 2020-07-17T12:02:19.084 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 0 2020-07-17T12:02:19.085 INFO:teuthology.orchestra.run.smithi052.stdout:fsid 4481a3ce-4482-458d-bce8-7b9423f8702f 2020-07-17T12:02:19.085 INFO:teuthology.orchestra.run.smithi052.stdout:last_changed 2020-07-17T12:02:19.079581+0000 2020-07-17T12:02:19.085 INFO:teuthology.orchestra.run.smithi052.stdout:created 2020-07-17T12:02:19.079581+0000 2020-07-17T12:02:19.085 INFO:teuthology.orchestra.run.smithi052.stdout:min_mon_release 0 (unknown) 2020-07-17T12:02:19.086 INFO:teuthology.orchestra.run.smithi052.stdout:0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2020-07-17T12:02:19.086 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-07-17T12:02:19.087 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 4481a3ce-4482-458d-bce8-7b9423f8702f... 2020-07-17T12:02:19.089 INFO:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-07-17T12:02:19.133 INFO:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-07-17T12:02:19.165 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2020-07-17T12:02:19.166 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2020-07-17T12:02:19.166 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-17T12:02:19.166 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2020-07-17T12:02:19.167 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.167 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2020-07-17T12:02:19.167 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.167 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2020-07-17T12:02:19.167 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.168 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2020-07-17T12:02:19.168 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2020-07-17T12:02:19.168 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.168 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2020-07-17T12:02:19.168 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2020-07-17T12:02:19.169 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.170 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-17T12:02:19.170 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2020-07-17T12:02:19.170 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.170 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2020-07-17T12:02:19.171 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-17T12:02:19.171 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.171 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2020-07-17T12:02:19.171 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.171 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2020-07-17T12:02:19.172 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2020-07-17T12:02:19.172 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2020-07-17T12:02:19.172 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2020-07-17T12:02:19.172 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2020-07-17T12:02:19.172 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2020-07-17T12:02:19.172 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2020-07-17T12:02:19.173 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-17T12:02:19.173 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-17T12:02:19.173 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2020-07-17T12:02:19.173 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.173 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-17T12:02:19.174 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.174 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2020-07-17T12:02:19.174 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.175 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2020-07-17T12:02:19.175 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.175 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2020-07-17T12:02:19.175 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2020-07-17T12:02:19.175 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = 172.21.15.52 2020-07-17T12:02:19.176 INFO:teuthology.orchestra.run.smithi052.stdout: osd heartbeat grace = 40 2020-07-17T12:02:19.176 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 4481a3ce-4482-458d-bce8-7b9423f8702f 2020-07-17T12:02:19.176 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.176 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2020-07-17T12:02:19.177 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2020-07-17T12:02:19.177 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.177 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2020-07-17T12:02:19.177 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2020-07-17T12:02:19.177 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.178 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2020-07-17T12:02:19.178 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2020-07-17T12:02:19.178 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.179 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2020-07-17T12:02:19.179 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2020-07-17T12:02:19.179 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2020-07-17T12:02:19.180 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.180 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2020-07-17T12:02:19.181 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2020-07-17T12:02:19.181 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.181 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2020-07-17T12:02:19.181 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.181 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2020-07-17T12:02:19.182 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.182 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2020-07-17T12:02:19.182 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.182 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2020-07-17T12:02:19.182 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2020-07-17T12:02:19.183 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.183 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2020-07-17T12:02:19.183 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2020-07-17T12:02:19.183 INFO:teuthology.orchestra.run.smithi052.stdout: debug filestore = 20 2020-07-17T12:02:19.183 INFO:teuthology.orchestra.run.smithi052.stdout: debug journal = 20 2020-07-17T12:02:19.183 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 20 2020-07-17T12:02:19.184 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 25 2020-07-17T12:02:19.184 INFO:teuthology.orchestra.run.smithi052.stdout: osd fast shutdown = False 2020-07-17T12:02:19.184 INFO:teuthology.orchestra.run.smithi052.stdout: rocksdb delete range threshold = 0 2020-07-17T12:02:19.184 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.184 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2020-07-17T12:02:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2020-07-17T12:02:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2020-07-17T12:02:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2020-07-17T12:02:19.185 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2020-07-17T12:02:19.186 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2020-07-17T12:02:19.186 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2020-07-17T12:02:19.186 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2020-07-17T12:02:19.186 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.187 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2020-07-17T12:02:19.187 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2020-07-17T12:02:19.187 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2020-07-17T12:02:19.187 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2020-07-17T12:02:19.188 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2020-07-17T12:02:19.188 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2020-07-17T12:02:19.188 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2020-07-17T12:02:19.188 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2020-07-17T12:02:19.189 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2020-07-17T12:02:19.189 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2020-07-17T12:02:19.189 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2020-07-17T12:02:19.189 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd crush smoke test = False 2020-07-17T12:02:19.190 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:19.190 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2020-07-17T12:02:19.191 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2020-07-17T12:02:19.191 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2020-07-17T12:02:19.191 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2020-07-17T12:02:19.192 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-17T12:02:19.193 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-17T12:02:19.193 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw = 20 2020-07-17T12:02:19.193 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt require ssl = False 2020-07-17T12:02:19.194 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms backend = testing 2020-07-17T12:02:19.194 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-17T12:02:19.194 INFO:teuthology.orchestra.run.smithi052.stdout: rgw curl low speed time = 300 2020-07-17T12:02:19.195 INFO:teuthology.orchestra.run.smithi052.stdout: rgw data log num shards = 4 2020-07-17T12:02:19.195 INFO:teuthology.orchestra.run.smithi052.stdout: rgw md log max shards = 4 2020-07-17T12:02:19.196 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync log trim interval = 0 2020-07-17T12:02:19.196 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync obj etag verify = True 2020-07-17T12:02:19.196 INFO:teuthology.orchestra.run.smithi052.stdout: setgroup = ceph 2020-07-17T12:02:19.196 INFO:teuthology.orchestra.run.smithi052.stdout: setuser = ceph 2020-07-17T12:02:19.196 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2020-07-17T12:02:19.197 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2020-07-17T12:02:19.197 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 10 2020-07-17T12:02:19.197 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2020-07-17T12:02:19.197 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 1000 2020-07-17T12:02:19.197 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2020-07-17T12:02:19.198 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 10000 2020-07-17T12:02:19.202 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2020-07-17T12:02:19.202 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2020-07-17T12:02:19.202 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-17T12:02:19.202 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2020-07-17T12:02:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2020-07-17T12:02:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2020-07-17T12:02:19.204 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.204 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2020-07-17T12:02:19.205 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2020-07-17T12:02:19.205 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.205 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2020-07-17T12:02:19.205 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2020-07-17T12:02:19.206 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.207 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-17T12:02:19.208 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2020-07-17T12:02:19.208 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.208 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2020-07-17T12:02:19.208 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-17T12:02:19.209 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.209 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2020-07-17T12:02:19.209 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.210 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2020-07-17T12:02:19.210 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2020-07-17T12:02:19.210 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2020-07-17T12:02:19.210 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2020-07-17T12:02:19.210 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2020-07-17T12:02:19.211 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2020-07-17T12:02:19.211 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2020-07-17T12:02:19.211 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-17T12:02:19.211 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-17T12:02:19.212 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2020-07-17T12:02:19.212 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.212 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-17T12:02:19.212 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.213 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2020-07-17T12:02:19.213 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.213 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2020-07-17T12:02:19.214 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.214 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2020-07-17T12:02:19.214 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2020-07-17T12:02:19.214 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = 172.21.15.52 2020-07-17T12:02:19.214 INFO:teuthology.orchestra.run.smithi146.stdout: osd heartbeat grace = 40 2020-07-17T12:02:19.214 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 4481a3ce-4482-458d-bce8-7b9423f8702f 2020-07-17T12:02:19.215 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.215 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2020-07-17T12:02:19.215 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2020-07-17T12:02:19.216 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.216 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2020-07-17T12:02:19.216 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2020-07-17T12:02:19.216 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.216 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2020-07-17T12:02:19.217 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2020-07-17T12:02:19.217 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.217 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2020-07-17T12:02:19.217 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2020-07-17T12:02:19.217 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2020-07-17T12:02:19.218 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.218 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2020-07-17T12:02:19.218 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2020-07-17T12:02:19.218 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.218 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2020-07-17T12:02:19.218 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2020-07-17T12:02:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2020-07-17T12:02:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2020-07-17T12:02:19.221 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2020-07-17T12:02:19.221 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.221 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2020-07-17T12:02:19.221 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2020-07-17T12:02:19.221 INFO:teuthology.orchestra.run.smithi146.stdout: debug filestore = 20 2020-07-17T12:02:19.221 INFO:teuthology.orchestra.run.smithi146.stdout: debug journal = 20 2020-07-17T12:02:19.222 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 20 2020-07-17T12:02:19.222 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 25 2020-07-17T12:02:19.222 INFO:teuthology.orchestra.run.smithi146.stdout: osd fast shutdown = False 2020-07-17T12:02:19.222 INFO:teuthology.orchestra.run.smithi146.stdout: rocksdb delete range threshold = 0 2020-07-17T12:02:19.222 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.223 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2020-07-17T12:02:19.223 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2020-07-17T12:02:19.223 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2020-07-17T12:02:19.223 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2020-07-17T12:02:19.223 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2020-07-17T12:02:19.224 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2020-07-17T12:02:19.224 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2020-07-17T12:02:19.224 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2020-07-17T12:02:19.224 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.224 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2020-07-17T12:02:19.224 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2020-07-17T12:02:19.225 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2020-07-17T12:02:19.225 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2020-07-17T12:02:19.225 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2020-07-17T12:02:19.225 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2020-07-17T12:02:19.225 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2020-07-17T12:02:19.226 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2020-07-17T12:02:19.227 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2020-07-17T12:02:19.227 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2020-07-17T12:02:19.227 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2020-07-17T12:02:19.228 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd crush smoke test = False 2020-07-17T12:02:19.228 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:02:19.228 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2020-07-17T12:02:19.229 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2020-07-17T12:02:19.229 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2020-07-17T12:02:19.229 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2020-07-17T12:02:19.242 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-17T12:02:19.243 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-17T12:02:19.243 INFO:teuthology.orchestra.run.smithi146.stdout: debug rgw = 20 2020-07-17T12:02:19.243 INFO:teuthology.orchestra.run.smithi146.stdout: rgw crypt require ssl = False 2020-07-17T12:02:19.244 INFO:teuthology.orchestra.run.smithi146.stdout: rgw crypt s3 kms backend = testing 2020-07-17T12:02:19.244 INFO:teuthology.orchestra.run.smithi146.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-17T12:02:19.244 INFO:teuthology.orchestra.run.smithi146.stdout: rgw curl low speed time = 300 2020-07-17T12:02:19.244 INFO:teuthology.orchestra.run.smithi146.stdout: rgw data log num shards = 4 2020-07-17T12:02:19.245 INFO:teuthology.orchestra.run.smithi146.stdout: rgw md log max shards = 4 2020-07-17T12:02:19.245 INFO:teuthology.orchestra.run.smithi146.stdout: rgw sync log trim interval = 0 2020-07-17T12:02:19.245 INFO:teuthology.orchestra.run.smithi146.stdout: rgw sync obj etag verify = True 2020-07-17T12:02:19.256 INFO:teuthology.orchestra.run.smithi146.stdout: setgroup = ceph 2020-07-17T12:02:19.257 INFO:teuthology.orchestra.run.smithi146.stdout: setuser = ceph 2020-07-17T12:02:19.257 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2020-07-17T12:02:19.257 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2020-07-17T12:02:19.258 INFO:teuthology.orchestra.run.smithi146.stdout: osd_max_omap_entries_per_request = 10 2020-07-17T12:02:19.258 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2020-07-17T12:02:19.258 INFO:teuthology.orchestra.run.smithi146.stdout: osd_max_omap_entries_per_request = 1000 2020-07-17T12:02:19.259 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2020-07-17T12:02:19.259 INFO:teuthology.orchestra.run.smithi146.stdout: osd_max_omap_entries_per_request = 10000 2020-07-17T12:02:19.260 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-07-17T12:02:19.261 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-07-17T12:02:19.311 INFO:tasks.ceph:Copying monmap to all nodes... 2020-07-17T12:02:19.380 DEBUG:teuthology.orchestra.remote:smithi052:/etc/ceph/c2.keyring is 206B 2020-07-17T12:02:19.442 DEBUG:teuthology.orchestra.remote:smithi052:/home/ubuntu/cephtest/c2.monmap is 179B 2020-07-17T12:02:19.453 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2020-07-17T12:02:19.454 INFO:teuthology.orchestra.run.smithi146:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-07-17T12:02:19.511 INFO:teuthology.orchestra.run.smithi146:> cat > /home/ubuntu/cephtest/c2.monmap 2020-07-17T12:02:19.602 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2020-07-17T12:02:19.603 INFO:teuthology.orchestra.run.smithi052:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-07-17T12:02:19.629 INFO:teuthology.orchestra.run.smithi052:> cat > /home/ubuntu/cephtest/c2.monmap 2020-07-17T12:02:19.725 INFO:tasks.ceph:Setting up mon nodes... 2020-07-17T12:02:19.725 INFO:tasks.ceph:Setting up mgr nodes... 2020-07-17T12:02:19.726 INFO:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-07-17T12:02:19.812 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-07-17T12:02:19.815 INFO:tasks.ceph:Setting up mds nodes... 2020-07-17T12:02:19.816 INFO:tasks.ceph_client:Setting up client nodes... 2020-07-17T12:02:19.816 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-07-17T12:02:19.902 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/c2.client.0.keyring 2020-07-17T12:02:19.914 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-07-17T12:02:19.990 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/c2.client.1.keyring 2020-07-17T12:02:20.003 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-07-17T12:02:20.003 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-17T12:02:20.004 INFO:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-07-17T12:02:20.077 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-17T12:02:20.078 INFO:tasks.ceph:role: c2.osd.0 2020-07-17T12:02:20.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2020-07-17T12:02:20.079 INFO:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-07-17T12:02:20.159 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-07-17T12:02:20.159 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-17T12:02:20.160 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-17T12:02:20.160 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-17T12:02:20.160 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2020-07-17T12:02:20.160 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-17T12:02:20.161 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-17T12:02:20.161 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-17T12:02:20.161 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-17T12:02:20.162 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2020-07-17T12:02:20.162 INFO:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-07-17T12:02:20.224 INFO:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-07-17T12:02:20.279 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: /sbin/restorecon: command not found 2020-07-17T12:02:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-17T12:02:20.281 INFO:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-07-17T12:02:20.337 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-17T12:02:20.338 INFO:tasks.ceph:role: c2.osd.1 2020-07-17T12:02:20.338 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com 2020-07-17T12:02:20.338 INFO:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-07-17T12:02:20.420 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-07-17T12:02:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-17T12:02:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-17T12:02:20.421 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-17T12:02:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2020-07-17T12:02:20.421 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-17T12:02:20.421 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-17T12:02:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-17T12:02:20.422 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-17T12:02:20.423 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2020-07-17T12:02:20.423 INFO:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-07-17T12:02:20.488 INFO:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-07-17T12:02:20.553 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: /sbin/restorecon: command not found 2020-07-17T12:02:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-17T12:02:20.554 INFO:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-07-17T12:02:20.615 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-17T12:02:20.615 INFO:tasks.ceph:role: c2.osd.2 2020-07-17T12:02:20.616 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com 2020-07-17T12:02:20.616 INFO:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-07-17T12:02:20.702 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-07-17T12:02:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-17T12:02:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-17T12:02:20.703 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-17T12:02:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2020-07-17T12:02:20.703 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-17T12:02:20.703 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-17T12:02:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-17T12:02:20.704 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-17T12:02:20.704 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2020-07-17T12:02:20.705 INFO:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-07-17T12:02:20.769 INFO:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-07-17T12:02:20.823 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: /sbin/restorecon: command not found 2020-07-17T12:02:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-17T12:02:20.824 INFO:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-07-17T12:02:20.917 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:20.911+0000 7f80b6d76d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-07-17T12:02:20.917 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:20.911+0000 7f80b6d76d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-07-17T12:02:20.917 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:20.911+0000 7f80b6d76d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-07-17T12:02:20.917 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:20.911+0000 7f80b6d76d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-07-17T12:02:20.918 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:20.911+0000 7f80b6d76d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-07-17T12:02:20.918 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:20.911+0000 7f80b6d76d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-07-17T12:02:22.847 INFO:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-07-17T12:02:22.861 INFO:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-07-17T12:02:22.942 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:22.935+0000 7f4d11868d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-07-17T12:02:22.942 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:22.935+0000 7f4d11868d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-07-17T12:02:22.942 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:22.935+0000 7f4d11868d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-07-17T12:02:22.942 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:22.935+0000 7f4d11868d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-07-17T12:02:22.943 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:22.935+0000 7f4d11868d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-07-17T12:02:22.943 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:22.935+0000 7f4d11868d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-07-17T12:02:24.866 INFO:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-07-17T12:02:24.923 INFO:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-07-17T12:02:25.008 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:25.003+0000 7fa3a3e45d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-07-17T12:02:25.008 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:25.003+0000 7fa3a3e45d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-07-17T12:02:25.008 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:25.003+0000 7fa3a3e45d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-07-17T12:02:25.008 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:25.003+0000 7fa3a3e45d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-07-17T12:02:25.009 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:25.003+0000 7fa3a3e45d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-07-17T12:02:25.009 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:02:25.003+0000 7fa3a3e45d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-07-17T12:02:26.938 INFO:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-07-17T12:02:26.995 INFO:tasks.ceph:Reading keys from all nodes... 2020-07-17T12:02:26.996 INFO:teuthology.orchestra.run.smithi052:> mktemp 2020-07-17T12:02:27.044 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.t0rZ5EClJY 2020-07-17T12:02:27.045 INFO:teuthology.orchestra.run.smithi052:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.t0rZ5EClJY 2020-07-17T12:02:27.103 INFO:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.t0rZ5EClJY 2020-07-17T12:02:27.211 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.t0rZ5EClJY is 56B 2020-07-17T12:02:27.222 INFO:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.t0rZ5EClJY 2020-07-17T12:02:27.228 INFO:teuthology.orchestra.run.smithi052:> mktemp 2020-07-17T12:02:27.276 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.WOjAtEnPBP 2020-07-17T12:02:27.277 INFO:teuthology.orchestra.run.smithi052:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.WOjAtEnPBP 2020-07-17T12:02:27.334 INFO:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.WOjAtEnPBP 2020-07-17T12:02:27.439 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.WOjAtEnPBP is 56B 2020-07-17T12:02:27.449 INFO:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.WOjAtEnPBP 2020-07-17T12:02:27.455 INFO:teuthology.orchestra.run.smithi052:> mktemp 2020-07-17T12:02:27.504 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.TiuiRPZwsH 2020-07-17T12:02:27.505 INFO:teuthology.orchestra.run.smithi052:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.TiuiRPZwsH 2020-07-17T12:02:27.562 INFO:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.TiuiRPZwsH 2020-07-17T12:02:27.671 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.TiuiRPZwsH is 56B 2020-07-17T12:02:27.720 INFO:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.TiuiRPZwsH 2020-07-17T12:02:27.727 INFO:teuthology.orchestra.run.smithi052:> mktemp 2020-07-17T12:02:27.773 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.OuPgHOGz9p 2020-07-17T12:02:27.773 INFO:teuthology.orchestra.run.smithi052:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.OuPgHOGz9p 2020-07-17T12:02:27.830 INFO:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.OuPgHOGz9p 2020-07-17T12:02:27.939 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.OuPgHOGz9p is 56B 2020-07-17T12:02:27.950 INFO:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.OuPgHOGz9p 2020-07-17T12:02:28.007 DEBUG:teuthology.orchestra.remote:smithi052:/etc/ceph/c2.client.0.keyring is 59B 2020-07-17T12:02:28.025 DEBUG:teuthology.orchestra.remote:smithi052:/etc/ceph/c2.client.1.keyring is 59B 2020-07-17T12:02:28.080 INFO:tasks.ceph:Adding keys to all mons... 2020-07-17T12:02:28.081 INFO:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/c2.keyring 2020-07-17T12:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2020-07-17T12:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQBLkxFfKWY0MBAAROOgv+aZ6c2+D2XUMy1w2A== 2020-07-17T12:02:28.096 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2020-07-17T12:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQBMkxFfJGxhNhAASDLYvwOMdGKfXkpP2YDpLg== 2020-07-17T12:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2020-07-17T12:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQBOkxFfbQflNxAA9pVTKDVHv8K/ZiwjpVuxbw== 2020-07-17T12:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2020-07-17T12:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQBRkxFf/Jc4ABAACpx/qWHz7HAZLGl3e0TMew== 2020-07-17T12:02:28.097 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2020-07-17T12:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQBLkxFfNj+FNRAApBQj5SxlltUN76mrRol/mA== 2020-07-17T12:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout:[client.1] 2020-07-17T12:02:28.098 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQBLkxFfHpjJOhAAxUlh1IQd/f6IxHZWQMItTw== 2020-07-17T12:02:28.099 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-07-17T12:02:28.185 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-17T12:02:28.269 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-17T12:02:28.358 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-17T12:02:28.442 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-17T12:02:28.539 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-17T12:02:28.582 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-07-17T12:02:28.582 INFO:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-07-17T12:02:28.642 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-07-17T12:02:28.735 INFO:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-07-17T12:02:28.749 INFO:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-07-17T12:02:28.797 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-07-17T12:02:28.798 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-07-17T12:02:28.799 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-07-17T12:02:28.841 INFO:tasks.ceph.c2.mon.a:Started 2020-07-17T12:02:28.842 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-07-17T12:02:28.843 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-07-17T12:02:28.844 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-07-17T12:02:28.846 INFO:tasks.ceph.c2.mgr.x:Started 2020-07-17T12:02:28.847 INFO:tasks.ceph:Setting crush tunables to default 2020-07-17T12:02:28.847 INFO:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd crush tunables default 2020-07-17T12:02:29.312 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2020-07-17T12:02:29.319 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-07-17T12:02:29.320 INFO:teuthology.orchestra.run.smithi052:> mktemp 2020-07-17T12:02:29.326 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.48XbtMef8v 2020-07-17T12:02:29.327 INFO:teuthology.orchestra.run.smithi052:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.48XbtMef8v 2020-07-17T12:02:29.381 INFO:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.48XbtMef8v 2020-07-17T12:02:29.486 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.48XbtMef8v is 37B 2020-07-17T12:02:29.497 INFO:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.48XbtMef8v 2020-07-17T12:02:29.502 INFO:teuthology.orchestra.run.smithi052:> mktemp 2020-07-17T12:02:29.548 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.0pPBIySlhi 2020-07-17T12:02:29.549 INFO:teuthology.orchestra.run.smithi052:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.0pPBIySlhi 2020-07-17T12:02:29.605 INFO:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.0pPBIySlhi 2020-07-17T12:02:29.718 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.0pPBIySlhi is 37B 2020-07-17T12:02:29.769 INFO:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.0pPBIySlhi 2020-07-17T12:02:29.775 INFO:teuthology.orchestra.run.smithi052:> mktemp 2020-07-17T12:02:29.824 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.QlRhx4sdFy 2020-07-17T12:02:29.824 INFO:teuthology.orchestra.run.smithi052:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.QlRhx4sdFy 2020-07-17T12:02:29.882 INFO:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.QlRhx4sdFy 2020-07-17T12:02:29.987 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.QlRhx4sdFy is 37B 2020-07-17T12:02:30.036 INFO:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.QlRhx4sdFy 2020-07-17T12:02:30.042 INFO:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd new 622656c7-376b-431e-94fd-e0e2a991d437 0 2020-07-17T12:02:30.284 INFO:teuthology.orchestra.run.smithi052.stdout:0 2020-07-17T12:02:30.300 INFO:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd new b5a5ff61-b50c-48db-94ad-325b91020283 1 2020-07-17T12:02:30.497 INFO:teuthology.orchestra.run.smithi052.stdout:1 2020-07-17T12:02:30.513 INFO:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd new 1ea53c99-c138-42f2-b0c4-e51e9ff57661 2 2020-07-17T12:02:30.705 INFO:teuthology.orchestra.run.smithi052.stdout:2 2020-07-17T12:02:30.717 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-07-17T12:02:30.717 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-07-17T12:02:30.720 INFO:tasks.ceph.c2.osd.0:Started 2020-07-17T12:02:30.720 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-07-17T12:02:30.720 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-07-17T12:02:30.722 INFO:tasks.ceph.c2.osd.1:Started 2020-07-17T12:02:30.723 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-07-17T12:02:30.724 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-07-17T12:02:30.726 INFO:tasks.ceph.c2.osd.2:Started 2020-07-17T12:02:30.727 INFO:tasks.ceph:Waiting for OSDs to come up 2020-07-17T12:02:31.030 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-07-17T12:02:31.092 INFO:tasks.ceph.c2.osd.0.smithi052.stderr:2020-07-17T12:02:31.087+0000 7f5c96981d80 -1 Falling back to public interface 2020-07-17T12:02:31.203 INFO:teuthology.misc.health.smithi052.stdout: 2020-07-17T12:02:31.203 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":5,"fsid":"4481a3ce-4482-458d-bce8-7b9423f8702f","created":"2020-07-17T12:02:28.948816+0000","modified":"2020-07-17T12:02:30.697705+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"622656c7-376b-431e-94fd-e0e2a991d437","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b5a5ff61-b50c-48db-94ad-325b91020283","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1ea53c99-c138-42f2-b0c4-e51e9ff57661","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:31.213 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-07-17T12:02:31.325 INFO:tasks.ceph.c2.osd.1.smithi052.stderr:2020-07-17T12:02:31.319+0000 7fdc73547d80 -1 Falling back to public interface 2020-07-17T12:02:31.365 INFO:tasks.ceph.c2.osd.2.smithi052.stderr:2020-07-17T12:02:31.359+0000 7fd82b4d2d80 -1 Falling back to public interface 2020-07-17T12:02:31.700 INFO:tasks.ceph.c2.osd.0.smithi052.stderr:2020-07-17T12:02:31.695+0000 7f5c96981d80 -1 osd.0 0 log_to_monitors {default=true} 2020-07-17T12:02:31.933 INFO:tasks.ceph.c2.osd.1.smithi052.stderr:2020-07-17T12:02:31.927+0000 7fdc73547d80 -1 osd.1 0 log_to_monitors {default=true} 2020-07-17T12:02:32.001 INFO:tasks.ceph.c2.osd.2.smithi052.stderr:2020-07-17T12:02:31.995+0000 7fd82b4d2d80 -1 osd.2 0 log_to_monitors {default=true} 2020-07-17T12:02:32.984 INFO:tasks.ceph.c2.osd.0.smithi052.stderr:2020-07-17T12:02:32.979+0000 7f5c8de4c700 -1 osd.0 0 waiting for initial osdmap 2020-07-17T12:02:32.984 INFO:tasks.ceph.c2.osd.1.smithi052.stderr:2020-07-17T12:02:32.979+0000 7fdc6aa12700 -1 osd.1 0 waiting for initial osdmap 2020-07-17T12:02:33.992 INFO:tasks.ceph.c2.osd.2.smithi052.stderr:2020-07-17T12:02:33.987+0000 7fd82299d700 -1 osd.2 0 waiting for initial osdmap 2020-07-17T12:02:37.517 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-07-17T12:02:37.785 INFO:teuthology.misc.health.smithi052.stdout: 2020-07-17T12:02:37.786 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"4481a3ce-4482-458d-bce8-7b9423f8702f","created":"2020-07-17T12:02:28.948816+0000","modified":"2020-07-17T12:02:36.039121+0000","last_up_change":"2020-07-17T12:02:34.981146+0000","last_in_change":"2020-07-17T12:02:34.981146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-17T12:02:32.807214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"622656c7-376b-431e-94fd-e0e2a991d437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":11764},{"type":"v1","addr":"172.21.15.52:6801","nonce":11764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":11764},{"type":"v1","addr":"172.21.15.52:6803","nonce":11764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":11764},{"type":"v1","addr":"172.21.15.52:6807","nonce":11764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":11764},{"type":"v1","addr":"172.21.15.52:6805","nonce":11764}]},"public_addr":"172.21.15.52:6801/11764","cluster_addr":"172.21.15.52:6803/11764","heartbeat_back_addr":"172.21.15.52:6807/11764","heartbeat_front_addr":"172.21.15.52:6805/11764","state":["exists","up"]},{"osd":1,"uuid":"b5a5ff61-b50c-48db-94ad-325b91020283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":11765},{"type":"v1","addr":"172.21.15.52:6809","nonce":11765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":11765},{"type":"v1","addr":"172.21.15.52:6811","nonce":11765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":11765},{"type":"v1","addr":"172.21.15.52:6815","nonce":11765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":11765},{"type":"v1","addr":"172.21.15.52:6813","nonce":11765}]},"public_addr":"172.21.15.52:6809/11765","cluster_addr":"172.21.15.52:6811/11765","heartbeat_back_addr":"172.21.15.52:6815/11765","heartbeat_front_addr":"172.21.15.52:6813/11765","state":["exists","up"]},{"osd":2,"uuid":"1ea53c99-c138-42f2-b0c4-e51e9ff57661","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":11774},{"type":"v1","addr":"172.21.15.52:6817","nonce":11774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":11774},{"type":"v1","addr":"172.21.15.52:6819","nonce":11774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":11774},{"type":"v1","addr":"172.21.15.52:6823","nonce":11774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":11774},{"type":"v1","addr":"172.21.15.52:6821","nonce":11774}]},"public_addr":"172.21.15.52:6817/11774","cluster_addr":"172.21.15.52:6819/11774","heartbeat_back_addr":"172.21.15.52:6823/11774","heartbeat_front_addr":"172.21.15.52:6821/11774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:32.671125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:32.886081+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:33.017148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:37.798 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-07-17T12:02:37.798 INFO:tasks.ceph:Creating RBD pool 2020-07-17T12:02:37.799 INFO:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd pool create rbd 8 2020-07-17T12:02:39.072 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'rbd' created 2020-07-17T12:02:39.084 INFO:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-07-17T12:02:40.078 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rbd' on pool 'rbd' 2020-07-17T12:02:40.095 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-07-17T12:02:40.096 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-07-17T12:02:40.097 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-07-17T12:02:40.358 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:40.359 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":12,"fsid":"4481a3ce-4482-458d-bce8-7b9423f8702f","created":"2020-07-17T12:02:28.948816+0000","modified":"2020-07-17T12:02:40.055155+0000","last_up_change":"2020-07-17T12:02:34.981146+0000","last_in_change":"2020-07-17T12:02:34.981146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-17T12:02:32.807214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-17T12:02:38.065547+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"622656c7-376b-431e-94fd-e0e2a991d437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":11764},{"type":"v1","addr":"172.21.15.52:6801","nonce":11764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":11764},{"type":"v1","addr":"172.21.15.52:6803","nonce":11764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":11764},{"type":"v1","addr":"172.21.15.52:6807","nonce":11764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":11764},{"type":"v1","addr":"172.21.15.52:6805","nonce":11764}]},"public_addr":"172.21.15.52:6801/11764","cluster_addr":"172.21.15.52:6803/11764","heartbeat_back_addr":"172.21.15.52:6807/11764","heartbeat_front_addr":"172.21.15.52:6805/11764","state":["exists","up"]},{"osd":1,"uuid":"b5a5ff61-b50c-48db-94ad-325b91020283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":11765},{"type":"v1","addr":"172.21.15.52:6809","nonce":11765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":11765},{"type":"v1","addr":"172.21.15.52:6811","nonce":11765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":11765},{"type":"v1","addr":"172.21.15.52:6815","nonce":11765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":11765},{"type":"v1","addr":"172.21.15.52:6813","nonce":11765}]},"public_addr":"172.21.15.52:6809/11765","cluster_addr":"172.21.15.52:6811/11765","heartbeat_back_addr":"172.21.15.52:6815/11765","heartbeat_front_addr":"172.21.15.52:6813/11765","state":["exists","up"]},{"osd":2,"uuid":"1ea53c99-c138-42f2-b0c4-e51e9ff57661","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":11774},{"type":"v1","addr":"172.21.15.52:6817","nonce":11774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":11774},{"type":"v1","addr":"172.21.15.52:6819","nonce":11774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":11774},{"type":"v1","addr":"172.21.15.52:6823","nonce":11774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":11774},{"type":"v1","addr":"172.21.15.52:6821","nonce":11774}]},"public_addr":"172.21.15.52:6817/11774","cluster_addr":"172.21.15.52:6819/11774","heartbeat_back_addr":"172.21.15.52:6823/11774","heartbeat_front_addr":"172.21.15.52:6821/11774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:32.671125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:32.886081+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:33.017148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:40.372 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-07-17T12:02:32.807214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-07-17T12:02:38.065547+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-07-17T12:02:40.373 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2020-07-17T12:02:40.647 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2020-07-17T12:02:40.660 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2020-07-17T12:02:40.932 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 8 2020-07-17T12:02:40.946 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-07-17T12:02:40.946 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-07-17T12:02:40.946 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2020-07-17T12:02:41.259 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:41.272 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":11562},{"type":"v1","addr":"172.21.15.52:6825","nonce":11562}]},"active_addr":"172.21.15.52:6825/11562","active_change":"2020-07-17T12:02:32.761325+0000","active_mgr_features":4540138292837744639,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1201853244}]},{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1074143519}]},{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1168543235}]}]}} 2020-07-17T12:02:41.275 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-07-17T12:02:41.275 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-07-17T12:02:41.276 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-07-17T12:02:41.547 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:41.548 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":13,"fsid":"4481a3ce-4482-458d-bce8-7b9423f8702f","created":"2020-07-17T12:02:28.948816+0000","modified":"2020-07-17T12:02:41.061214+0000","last_up_change":"2020-07-17T12:02:34.981146+0000","last_in_change":"2020-07-17T12:02:34.981146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-17T12:02:32.807214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-17T12:02:38.065547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"622656c7-376b-431e-94fd-e0e2a991d437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":11764},{"type":"v1","addr":"172.21.15.52:6801","nonce":11764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":11764},{"type":"v1","addr":"172.21.15.52:6803","nonce":11764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":11764},{"type":"v1","addr":"172.21.15.52:6807","nonce":11764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":11764},{"type":"v1","addr":"172.21.15.52:6805","nonce":11764}]},"public_addr":"172.21.15.52:6801/11764","cluster_addr":"172.21.15.52:6803/11764","heartbeat_back_addr":"172.21.15.52:6807/11764","heartbeat_front_addr":"172.21.15.52:6805/11764","state":["exists","up"]},{"osd":1,"uuid":"b5a5ff61-b50c-48db-94ad-325b91020283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":11765},{"type":"v1","addr":"172.21.15.52:6809","nonce":11765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":11765},{"type":"v1","addr":"172.21.15.52:6811","nonce":11765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":11765},{"type":"v1","addr":"172.21.15.52:6815","nonce":11765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":11765},{"type":"v1","addr":"172.21.15.52:6813","nonce":11765}]},"public_addr":"172.21.15.52:6809/11765","cluster_addr":"172.21.15.52:6811/11765","heartbeat_back_addr":"172.21.15.52:6815/11765","heartbeat_front_addr":"172.21.15.52:6813/11765","state":["exists","up"]},{"osd":2,"uuid":"1ea53c99-c138-42f2-b0c4-e51e9ff57661","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":11774},{"type":"v1","addr":"172.21.15.52:6817","nonce":11774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":11774},{"type":"v1","addr":"172.21.15.52:6819","nonce":11774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":11774},{"type":"v1","addr":"172.21.15.52:6823","nonce":11774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":11774},{"type":"v1","addr":"172.21.15.52:6821","nonce":11774}]},"public_addr":"172.21.15.52:6817/11774","cluster_addr":"172.21.15.52:6819/11774","heartbeat_back_addr":"172.21.15.52:6823/11774","heartbeat_front_addr":"172.21.15.52:6821/11774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:32.671125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:32.886081+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:33.017148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:41.567 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-07-17T12:02:41.568 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-07-17T12:02:41.837 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:41.837 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":13,"fsid":"4481a3ce-4482-458d-bce8-7b9423f8702f","created":"2020-07-17T12:02:28.948816+0000","modified":"2020-07-17T12:02:41.061214+0000","last_up_change":"2020-07-17T12:02:34.981146+0000","last_in_change":"2020-07-17T12:02:34.981146+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-17T12:02:32.807214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-17T12:02:38.065547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"622656c7-376b-431e-94fd-e0e2a991d437","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":11764},{"type":"v1","addr":"172.21.15.52:6801","nonce":11764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":11764},{"type":"v1","addr":"172.21.15.52:6803","nonce":11764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":11764},{"type":"v1","addr":"172.21.15.52:6807","nonce":11764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":11764},{"type":"v1","addr":"172.21.15.52:6805","nonce":11764}]},"public_addr":"172.21.15.52:6801/11764","cluster_addr":"172.21.15.52:6803/11764","heartbeat_back_addr":"172.21.15.52:6807/11764","heartbeat_front_addr":"172.21.15.52:6805/11764","state":["exists","up"]},{"osd":1,"uuid":"b5a5ff61-b50c-48db-94ad-325b91020283","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":11765},{"type":"v1","addr":"172.21.15.52:6809","nonce":11765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":11765},{"type":"v1","addr":"172.21.15.52:6811","nonce":11765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":11765},{"type":"v1","addr":"172.21.15.52:6815","nonce":11765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":11765},{"type":"v1","addr":"172.21.15.52:6813","nonce":11765}]},"public_addr":"172.21.15.52:6809/11765","cluster_addr":"172.21.15.52:6811/11765","heartbeat_back_addr":"172.21.15.52:6815/11765","heartbeat_front_addr":"172.21.15.52:6813/11765","state":["exists","up"]},{"osd":2,"uuid":"1ea53c99-c138-42f2-b0c4-e51e9ff57661","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":11774},{"type":"v1","addr":"172.21.15.52:6817","nonce":11774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":11774},{"type":"v1","addr":"172.21.15.52:6819","nonce":11774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":11774},{"type":"v1","addr":"172.21.15.52:6823","nonce":11774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":11774},{"type":"v1","addr":"172.21.15.52:6821","nonce":11774}]},"public_addr":"172.21.15.52:6817/11774","cluster_addr":"172.21.15.52:6819/11774","heartbeat_back_addr":"172.21.15.52:6823/11774","heartbeat_front_addr":"172.21.15.52:6821/11774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:32.671125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:32.886081+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-07-17T12:02:33.017148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-17T12:02:41.851 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2020-07-17T12:02:42.012 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2020-07-17T12:02:42.013 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2020-07-17T12:02:42.175 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2020-07-17T12:02:42.176 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2020-07-17T12:02:42.337 INFO:teuthology.orchestra.run.smithi052.stdout:38654705668 2020-07-17T12:02:42.338 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-07-17T12:02:42.604 INFO:teuthology.orchestra.run.smithi052.stdout:34359738370 2020-07-17T12:02:42.616 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2020-07-17T12:02:43.618 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-07-17T12:02:43.899 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2020-07-17T12:02:43.912 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2020-07-17T12:02:43.913 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-07-17T12:02:44.185 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2020-07-17T12:02:44.199 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2020-07-17T12:02:44.199 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2020-07-17T12:02:44.470 INFO:teuthology.orchestra.run.smithi052.stdout:38654705668 2020-07-17T12:02:44.484 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.2 2020-07-17T12:02:44.484 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-07-17T12:02:44.484 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-07-17T12:02:44.748 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:44.751 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2020-07-17T12:02:44.761 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-07-17T12:02:42.772246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146076,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426724,"statfs":{"total":322122547200,"available":318900965376,"internally_reserved":3221225472,"allocated":331776,"data_stored":96184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.710820"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.078338+0000","last_change":"2020-07-17T12:02:40.077041+0000","last_active":"2020-07-17T12:02:40.078338+0000","last_peered":"2020-07-17T12:02:40.078338+0000","last_clean":"2020-07-17T12:02:40.078338+0000","last_became_active":"2020-07-17T12:02:40.076573+0000","last_became_peered":"2020-07-17T12:02:40.076573+0000","last_unstale":"2020-07-17T12:02:40.078338+0000","last_undegraded":"2020-07-17T12:02:40.078338+0000","last_fullsized":"2020-07-17T12:02:40.078338+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.078211+0000","last_change":"2020-07-17T12:02:40.077255+0000","last_active":"2020-07-17T12:02:40.078211+0000","last_peered":"2020-07-17T12:02:40.078211+0000","last_clean":"2020-07-17T12:02:40.078211+0000","last_became_active":"2020-07-17T12:02:40.076847+0000","last_became_peered":"2020-07-17T12:02:40.076847+0000","last_unstale":"2020-07-17T12:02:40.078211+0000","last_undegraded":"2020-07-17T12:02:40.078211+0000","last_fullsized":"2020-07-17T12:02:40.078211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.078357+0000","last_change":"2020-07-17T12:02:40.077730+0000","last_active":"2020-07-17T12:02:40.078357+0000","last_peered":"2020-07-17T12:02:40.078357+0000","last_clean":"2020-07-17T12:02:40.078357+0000","last_became_active":"2020-07-17T12:02:40.077406+0000","last_became_peered":"2020-07-17T12:02:40.077406+0000","last_unstale":"2020-07-17T12:02:40.078357+0000","last_undegraded":"2020-07-17T12:02:40.078357+0000","last_fullsized":"2020-07-17T12:02:40.078357+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.073002+0000","last_change":"2020-07-17T12:02:40.073002+0000","last_active":"2020-07-17T12:02:40.073002+0000","last_peered":"2020-07-17T12:02:40.073002+0000","last_clean":"2020-07-17T12:02:40.073002+0000","last_became_active":"2020-07-17T12:02:40.072483+0000","last_became_peered":"2020-07-17T12:02:40.072483+0000","last_unstale":"2020-07-17T12:02:40.073002+0000","last_undegraded":"2020-07-17T12:02:40.073002+0000","last_fullsized":"2020-07-17T12:02:40.073002+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.069490+0000","last_change":"2020-07-17T12:02:35.742852+0000","last_active":"2020-07-17T12:02:40.069490+0000","last_peered":"2020-07-17T12:02:40.069490+0000","last_clean":"2020-07-17T12:02:40.069490+0000","last_became_active":"2020-07-17T12:02:35.741290+0000","last_became_peered":"2020-07-17T12:02:35.741290+0000","last_unstale":"2020-07-17T12:02:40.069490+0000","last_undegraded":"2020-07-17T12:02:40.069490+0000","last_fullsized":"2020-07-17T12:02:40.069490+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:32.963995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:32.963995+0000","last_clean_scrub_stamp":"2020-07-17T12:02:32.963995+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.076507+0000","last_change":"2020-07-17T12:02:40.075436+0000","last_active":"2020-07-17T12:02:40.076507+0000","last_peered":"2020-07-17T12:02:40.076507+0000","last_clean":"2020-07-17T12:02:40.076507+0000","last_became_active":"2020-07-17T12:02:40.074060+0000","last_became_peered":"2020-07-17T12:02:40.074060+0000","last_unstale":"2020-07-17T12:02:40.076507+0000","last_undegraded":"2020-07-17T12:02:40.076507+0000","last_fullsized":"2020-07-17T12:02:40.076507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.073088+0000","last_change":"2020-07-17T12:02:40.073088+0000","last_active":"2020-07-17T12:02:40.073088+0000","last_peered":"2020-07-17T12:02:40.073088+0000","last_clean":"2020-07-17T12:02:40.073088+0000","last_became_active":"2020-07-17T12:02:40.072559+0000","last_became_peered":"2020-07-17T12:02:40.072559+0000","last_unstale":"2020-07-17T12:02:40.073088+0000","last_undegraded":"2020-07-17T12:02:40.073088+0000","last_fullsized":"2020-07-17T12:02:40.073088+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.079342+0000","last_change":"2020-07-17T12:02:40.078476+0000","last_active":"2020-07-17T12:02:40.079342+0000","last_peered":"2020-07-17T12:02:40.079342+0000","last_clean":"2020-07-17T12:02:40.079342+0000","last_became_active":"2020-07-17T12:02:40.078027+0000","last_became_peered":"2020-07-17T12:02:40.078027+0000","last_unstale":"2020-07-17T12:02:40.079342+0000","last_undegraded":"2020-07-17T12:02:40.079342+0000","last_fullsized":"2020-07-17T12:02:40.079342+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.076404+0000","last_change":"2020-07-17T12:02:40.074079+0000","last_active":"2020-07-17T12:02:40.076404+0000","last_peered":"2020-07-17T12:02:40.076404+0000","last_clean":"2020-07-17T12:02:40.076404+0000","last_became_active":"2020-07-17T12:02:40.073471+0000","last_became_peered":"2020-07-17T12:02:40.073471+0000","last_unstale":"2020-07-17T12:02:40.076404+0000","last_undegraded":"2020-07-17T12:02:40.076404+0000","last_fullsized":"2020-07-17T12:02:40.076404+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048684,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808916,"statfs":{"total":107374182400,"available":106300329984,"internally_reserved":1073741824,"allocated":102400,"data_stored":29334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.403}]}]}]}} 2020-07-17T12:02:44.762 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-07-17T12:02:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:45.028 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2020-07-17T12:02:45.039 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-07-17T12:02:44.772754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146076,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426724,"statfs":{"total":322122547200,"available":318900965376,"internally_reserved":3221225472,"allocated":331776,"data_stored":96184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.711328"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.078338+0000","last_change":"2020-07-17T12:02:40.077041+0000","last_active":"2020-07-17T12:02:40.078338+0000","last_peered":"2020-07-17T12:02:40.078338+0000","last_clean":"2020-07-17T12:02:40.078338+0000","last_became_active":"2020-07-17T12:02:40.076573+0000","last_became_peered":"2020-07-17T12:02:40.076573+0000","last_unstale":"2020-07-17T12:02:40.078338+0000","last_undegraded":"2020-07-17T12:02:40.078338+0000","last_fullsized":"2020-07-17T12:02:40.078338+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.078211+0000","last_change":"2020-07-17T12:02:40.077255+0000","last_active":"2020-07-17T12:02:40.078211+0000","last_peered":"2020-07-17T12:02:40.078211+0000","last_clean":"2020-07-17T12:02:40.078211+0000","last_became_active":"2020-07-17T12:02:40.076847+0000","last_became_peered":"2020-07-17T12:02:40.076847+0000","last_unstale":"2020-07-17T12:02:40.078211+0000","last_undegraded":"2020-07-17T12:02:40.078211+0000","last_fullsized":"2020-07-17T12:02:40.078211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.078357+0000","last_change":"2020-07-17T12:02:40.077730+0000","last_active":"2020-07-17T12:02:40.078357+0000","last_peered":"2020-07-17T12:02:40.078357+0000","last_clean":"2020-07-17T12:02:40.078357+0000","last_became_active":"2020-07-17T12:02:40.077406+0000","last_became_peered":"2020-07-17T12:02:40.077406+0000","last_unstale":"2020-07-17T12:02:40.078357+0000","last_undegraded":"2020-07-17T12:02:40.078357+0000","last_fullsized":"2020-07-17T12:02:40.078357+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.073002+0000","last_change":"2020-07-17T12:02:40.073002+0000","last_active":"2020-07-17T12:02:40.073002+0000","last_peered":"2020-07-17T12:02:40.073002+0000","last_clean":"2020-07-17T12:02:40.073002+0000","last_became_active":"2020-07-17T12:02:40.072483+0000","last_became_peered":"2020-07-17T12:02:40.072483+0000","last_unstale":"2020-07-17T12:02:40.073002+0000","last_undegraded":"2020-07-17T12:02:40.073002+0000","last_fullsized":"2020-07-17T12:02:40.073002+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.069490+0000","last_change":"2020-07-17T12:02:35.742852+0000","last_active":"2020-07-17T12:02:40.069490+0000","last_peered":"2020-07-17T12:02:40.069490+0000","last_clean":"2020-07-17T12:02:40.069490+0000","last_became_active":"2020-07-17T12:02:35.741290+0000","last_became_peered":"2020-07-17T12:02:35.741290+0000","last_unstale":"2020-07-17T12:02:40.069490+0000","last_undegraded":"2020-07-17T12:02:40.069490+0000","last_fullsized":"2020-07-17T12:02:40.069490+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:32.963995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:32.963995+0000","last_clean_scrub_stamp":"2020-07-17T12:02:32.963995+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.076507+0000","last_change":"2020-07-17T12:02:40.075436+0000","last_active":"2020-07-17T12:02:40.076507+0000","last_peered":"2020-07-17T12:02:40.076507+0000","last_clean":"2020-07-17T12:02:40.076507+0000","last_became_active":"2020-07-17T12:02:40.074060+0000","last_became_peered":"2020-07-17T12:02:40.074060+0000","last_unstale":"2020-07-17T12:02:40.076507+0000","last_undegraded":"2020-07-17T12:02:40.076507+0000","last_fullsized":"2020-07-17T12:02:40.076507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.073088+0000","last_change":"2020-07-17T12:02:40.073088+0000","last_active":"2020-07-17T12:02:40.073088+0000","last_peered":"2020-07-17T12:02:40.073088+0000","last_clean":"2020-07-17T12:02:40.073088+0000","last_became_active":"2020-07-17T12:02:40.072559+0000","last_became_peered":"2020-07-17T12:02:40.072559+0000","last_unstale":"2020-07-17T12:02:40.073088+0000","last_undegraded":"2020-07-17T12:02:40.073088+0000","last_fullsized":"2020-07-17T12:02:40.073088+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.079342+0000","last_change":"2020-07-17T12:02:40.078476+0000","last_active":"2020-07-17T12:02:40.079342+0000","last_peered":"2020-07-17T12:02:40.079342+0000","last_clean":"2020-07-17T12:02:40.079342+0000","last_became_active":"2020-07-17T12:02:40.078027+0000","last_became_peered":"2020-07-17T12:02:40.078027+0000","last_unstale":"2020-07-17T12:02:40.079342+0000","last_undegraded":"2020-07-17T12:02:40.079342+0000","last_fullsized":"2020-07-17T12:02:40.079342+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-17T12:02:40.076404+0000","last_change":"2020-07-17T12:02:40.074079+0000","last_active":"2020-07-17T12:02:40.076404+0000","last_peered":"2020-07-17T12:02:40.076404+0000","last_clean":"2020-07-17T12:02:40.076404+0000","last_became_active":"2020-07-17T12:02:40.073471+0000","last_became_peered":"2020-07-17T12:02:40.073471+0000","last_unstale":"2020-07-17T12:02:40.076404+0000","last_undegraded":"2020-07-17T12:02:40.076404+0000","last_fullsized":"2020-07-17T12:02:40.076404+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-17T12:02:39.049349+0000","last_clean_scrub_stamp":"2020-07-17T12:02:39.049349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048684,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808916,"statfs":{"total":107374182400,"available":106300329984,"internally_reserved":1073741824,"allocated":102400,"data_stored":29334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.403}]}]}]}} 2020-07-17T12:02:45.040 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-07-17T12:02:45.040 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-07-17T12:02:45.041 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-07-17T12:02:45.041 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2020-07-17T12:02:45.358 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-07-17T12:02:45.423 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-07-17T12:02:45.424 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-07-17T12:02:45.424 INFO:teuthology.run_tasks:Running task rgw... 2020-07-17T12:02:45.443 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-07-17T12:02:45.443 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-07-17T12:02:45.444 INFO:tasks.rgw:Creating data pools 2020-07-17T12:02:45.444 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-07-17T12:02:45.445 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-07-17T12:02:46.218 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'default.rgw.buckets.data' created 2020-07-17T12:02:46.226 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-07-17T12:02:47.230 INFO:teuthology.orchestra.run.smithi146.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-17T12:02:47.254 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-07-17T12:02:48.229 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'default.rgw.buckets.index' created 2020-07-17T12:02:48.245 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-07-17T12:02:49.250 INFO:teuthology.orchestra.run.smithi146.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-17T12:02:49.269 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-07-17T12:02:49.270 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-07-17T12:02:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'default.rgw.buckets.data' already exists 2020-07-17T12:02:49.529 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-07-17T12:02:50.248 INFO:teuthology.orchestra.run.smithi146.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-17T12:02:50.262 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-07-17T12:02:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'default.rgw.buckets.index' already exists 2020-07-17T12:02:50.553 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-07-17T12:02:51.256 INFO:teuthology.orchestra.run.smithi146.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-17T12:02:51.263 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-07-17T12:02:51.264 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-07-17T12:02:51.971 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.data' created 2020-07-17T12:02:51.977 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-07-17T12:02:52.998 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-17T12:02:53.013 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-07-17T12:02:53.973 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.index' created 2020-07-17T12:02:53.988 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-07-17T12:02:55.007 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-17T12:02:55.023 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-07-17T12:02:55.024 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-07-17T12:02:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.data' already exists 2020-07-17T12:02:55.293 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-07-17T12:02:55.976 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-17T12:02:55.990 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-07-17T12:02:56.272 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.index' already exists 2020-07-17T12:02:56.283 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-07-17T12:02:56.988 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-17T12:02:57.001 DEBUG:tasks.rgw:Pools created 2020-07-17T12:02:57.002 INFO:tasks.rgw:Configuring compression type = random 2020-07-17T12:02:57.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-07-17T12:02:57.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-07-17T12:02:57.003 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-07-17T12:02:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:02:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:02:57.051 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:57.048+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.074 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.068+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.075 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.068+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.079 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.072+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.079 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.072+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.080 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.076+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.080 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.076+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.081 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.076+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.100 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.101 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.102 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.103 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 10 cannot find current period zonegroup using local zonegroup 2020-07-17T12:02:59.104 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.104 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.104 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.104 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.104 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 10 Creating default zonegroup 2020-07-17T12:02:59.105 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 20 get_system_obj_state: rctx=0x7ffd1a0153e0 obj=:.pools.avail state=0x5589650caf40 s->prefetch_data=0 2020-07-17T12:02:59.105 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.096+0000 7f642da170c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-07-17T12:02:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.112+0000 7f642da170c0 10 WARNING: store->list_zones() returned r=-2 2020-07-17T12:02:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.112+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.118 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.112+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.125 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.120+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.126 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.120+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.126 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.120+0000 7f642da170c0 10 could not read realm id: (2) No such file or directory 2020-07-17T12:02:59.127 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.120+0000 7f642da170c0 10 WARNING: failed to set zone as default, r=-22 2020-07-17T12:02:59.127 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.120+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.127 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.120+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.135 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.136 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-17T12:02:59.136 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.137 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:02:59.137 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 20 zonegroup default 2020-07-17T12:02:59.137 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.138 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:02:59.138 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 10 Cannot find current period zone using local zone 2020-07-17T12:02:59.138 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 10 Using default name default 2020-07-17T12:02:59.139 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.128+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.139 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.132+0000 7f642da170c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:02:59.139 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.132+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:02:59.140 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.132+0000 7f642da170c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-17T12:02:59.141 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.132+0000 7f642da170c0 20 zone default found 2020-07-17T12:02:59.141 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:02:59.132+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:01.093 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:01.088+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:01.094 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:01.088+0000 7f642da170c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:01.097 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:01.092+0000 7f642da170c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:01.097 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:01.092+0000 7f642da170c0 20 started sync module instance, tier type = 2020-07-17T12:03:01.097 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:01.092+0000 7f642da170c0 20 started zone id=efbb56bb-7d12-40f5-a59d-201e34b42071 (name=default) with tier type = 2020-07-17T12:03:03.140 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 add_watcher() i=0 2020-07-17T12:03:03.141 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 add_watcher() i=1 2020-07-17T12:03:03.141 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 add_watcher() i=2 2020-07-17T12:03:03.141 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 add_watcher() i=3 2020-07-17T12:03:03.141 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 add_watcher() i=4 2020-07-17T12:03:03.142 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 add_watcher() i=5 2020-07-17T12:03:03.142 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 add_watcher() i=6 2020-07-17T12:03:03.143 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 add_watcher() i=7 2020-07-17T12:03:03.143 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 2 all 8 watchers are set, enabling cache 2020-07-17T12:03:03.143 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-17T12:03:03.143 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f642da170c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-17T12:03:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:03.136+0000 7f63ea7ec700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-17T12:03:07.118 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.111+0000 7f642da170c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-07-17T12:03:07.219 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.215+0000 7f63597fa700 20 reqs_thread_entry: start 2020-07-17T12:03:07.220 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.215+0000 7f633effd700 20 reqs_thread_entry: start 2020-07-17T12:03:07.220 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.215+0000 7f642da170c0 20 init_complete bucket index max shards: 11 2020-07-17T12:03:07.221 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.215+0000 7f633d7fa700 20 reqs_thread_entry: start 2020-07-17T12:03:07.248 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:03:07.248 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.243+0000 7f63ea7ec700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-17T12:03:07.251 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.243+0000 7f642da170c0 20 remove_watcher() i=0 2020-07-17T12:03:07.251 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.243+0000 7f642da170c0 2 removed watcher, disabling cache 2020-07-17T12:03:07.254 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.247+0000 7f642da170c0 20 remove_watcher() i=1 2020-07-17T12:03:07.257 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.251+0000 7f642da170c0 20 remove_watcher() i=2 2020-07-17T12:03:07.260 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.255+0000 7f642da170c0 20 remove_watcher() i=3 2020-07-17T12:03:07.262 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.255+0000 7f642da170c0 20 remove_watcher() i=4 2020-07-17T12:03:07.270 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.259+0000 7f642da170c0 20 remove_watcher() i=5 2020-07-17T12:03:07.270 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.263+0000 7f642da170c0 20 remove_watcher() i=6 2020-07-17T12:03:07.271 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.263+0000 7f642da170c0 20 remove_watcher() i=7 2020-07-17T12:03:07.278 DEBUG:tasks.util.rgw: json result: [] 2020-07-17T12:03:07.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-17T12:03:07.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-17T12:03:07.280 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-17T12:03:07.360 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:07.360 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:07.373 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.367+0000 7f6396a2d0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-17T12:03:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.367+0000 7f6396a2d0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-17T12:03:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.375+0000 7f6396a2d0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.375+0000 7f62c7fff700 20 reqs_thread_entry: start 2020-07-17T12:03:07.382 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.375+0000 7f6396a2d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:07.382 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.375+0000 7f6396a2d0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.375+0000 7f6396a2d0c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-17T12:03:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.375+0000 7f6396a2d0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.384 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.379+0000 7f6396a2d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.384 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.379+0000 7f6396a2d0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.384 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.379+0000 7f6396a2d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:07.385 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.379+0000 7f6396a2d0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.385 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.379+0000 7f6396a2d0c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:03:07.392 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:07.393 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "efbb56bb-7d12-40f5-a59d-201e34b42071", 2020-07-17T12:03:07.393 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default", 2020-07-17T12:03:07.393 INFO:teuthology.orchestra.run.smithi146.stdout: "domain_root": "default.rgw.meta:root", 2020-07-17T12:03:07.394 INFO:teuthology.orchestra.run.smithi146.stdout: "control_pool": "default.rgw.control", 2020-07-17T12:03:07.394 INFO:teuthology.orchestra.run.smithi146.stdout: "gc_pool": "default.rgw.log:gc", 2020-07-17T12:03:07.394 INFO:teuthology.orchestra.run.smithi146.stdout: "lc_pool": "default.rgw.log:lc", 2020-07-17T12:03:07.394 INFO:teuthology.orchestra.run.smithi146.stdout: "log_pool": "default.rgw.log", 2020-07-17T12:03:07.394 INFO:teuthology.orchestra.run.smithi146.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-07-17T12:03:07.395 INFO:teuthology.orchestra.run.smithi146.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-07-17T12:03:07.395 INFO:teuthology.orchestra.run.smithi146.stdout: "roles_pool": "default.rgw.meta:roles", 2020-07-17T12:03:07.395 INFO:teuthology.orchestra.run.smithi146.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-07-17T12:03:07.395 INFO:teuthology.orchestra.run.smithi146.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-07-17T12:03:07.395 INFO:teuthology.orchestra.run.smithi146.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-07-17T12:03:07.396 INFO:teuthology.orchestra.run.smithi146.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-07-17T12:03:07.396 INFO:teuthology.orchestra.run.smithi146.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-07-17T12:03:07.396 INFO:teuthology.orchestra.run.smithi146.stdout: "otp_pool": "default.rgw.otp", 2020-07-17T12:03:07.396 INFO:teuthology.orchestra.run.smithi146.stdout: "system_key": { 2020-07-17T12:03:07.396 INFO:teuthology.orchestra.run.smithi146.stdout: "access_key": "", 2020-07-17T12:03:07.396 INFO:teuthology.orchestra.run.smithi146.stdout: "secret_key": "" 2020-07-17T12:03:07.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:07.397 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_pools": [ 2020-07-17T12:03:07.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:07.397 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "default-placement", 2020-07-17T12:03:07.397 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:03:07.398 INFO:teuthology.orchestra.run.smithi146.stdout: "index_pool": "default.rgw.buckets.index", 2020-07-17T12:03:07.398 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": { 2020-07-17T12:03:07.398 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD": { 2020-07-17T12:03:07.398 INFO:teuthology.orchestra.run.smithi146.stdout: "data_pool": "default.rgw.buckets.data", 2020-07-17T12:03:07.398 INFO:teuthology.orchestra.run.smithi146.stdout: "compression_type": "random" 2020-07-17T12:03:07.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:07.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:07.399 INFO:teuthology.orchestra.run.smithi146.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-07-17T12:03:07.399 INFO:teuthology.orchestra.run.smithi146.stdout: "index_type": 0 2020-07-17T12:03:07.399 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:07.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:07.400 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:07.400 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "" 2020-07-17T12:03:07.401 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:07.402 DEBUG:tasks.util.rgw: json result: {'id': 'efbb56bb-7d12-40f5-a59d-201e34b42071', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-17T12:03:07.403 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-07-17T12:03:07.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-07-17T12:03:07.404 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-07-17T12:03:07.476 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:07.476 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:07.495 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.483+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.495 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.483+0000 7f9ee83650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.495 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.483+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.496 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.483+0000 7f9ee83650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.496 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.483+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.496 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.483+0000 7f9ee83650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.497 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.483+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.497 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.487+0000 7f9ee83650c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.507+0000 7f9ee83650c0 20 RGWRados::pool_iterate: got zone_info.efbb56bb-7d12-40f5-a59d-201e34b42071 2020-07-17T12:03:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.507+0000 7f9ee83650c0 20 RGWRados::pool_iterate: got zonegroup_info.ed09f048-b1e4-4825-a7dd-fb0aff4a378c 2020-07-17T12:03:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.507+0000 7f9ee83650c0 20 RGWRados::pool_iterate: got zone_names.default 2020-07-17T12:03:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.507+0000 7f9ee83650c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-07-17T12:03:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.511+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.518 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.511+0000 7f9ee83650c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:07.519 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.511+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.519 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.511+0000 7f9ee83650c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-17T12:03:07.519 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.511+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.520 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.511+0000 7f9ee83650c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:07.520 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.511+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.521 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.511+0000 7f9ee83650c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:03:07.522 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.522 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.522 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 10 cannot find current period zonegroup using local zonegroup 2020-07-17T12:03:07.523 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.523 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.523 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.525 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:07.525 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.525 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:03:07.526 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 zonegroup default 2020-07-17T12:03:07.526 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.515+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.526 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.526 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 10 Cannot find current period zone using local zone 2020-07-17T12:03:07.527 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.527 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.527 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.529 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:07.529 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.537 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-17T12:03:07.537 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 20 zone default found 2020-07-17T12:03:07.538 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.519+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.538 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.523+0000 7f9ee83650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.538 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.523+0000 7f9ee83650c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.538 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.523+0000 7f9ee83650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.538 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.523+0000 7f9ee83650c0 20 started sync module instance, tier type = 2020-07-17T12:03:07.538 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.523+0000 7f9ee83650c0 20 started zone id=efbb56bb-7d12-40f5-a59d-201e34b42071 (name=default) with tier type = 2020-07-17T12:03:07.551 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.543+0000 7f9ee83650c0 20 add_watcher() i=0 2020-07-17T12:03:07.552 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.543+0000 7f9ee83650c0 20 add_watcher() i=1 2020-07-17T12:03:07.552 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.543+0000 7f9ee83650c0 20 add_watcher() i=2 2020-07-17T12:03:07.552 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.543+0000 7f9ee83650c0 20 add_watcher() i=3 2020-07-17T12:03:07.552 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.543+0000 7f9ee83650c0 20 add_watcher() i=4 2020-07-17T12:03:07.553 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.543+0000 7f9ee83650c0 20 add_watcher() i=5 2020-07-17T12:03:07.553 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.543+0000 7f9ee83650c0 20 add_watcher() i=6 2020-07-17T12:03:07.553 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.547+0000 7f9ee83650c0 20 add_watcher() i=7 2020-07-17T12:03:07.554 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.547+0000 7f9ee83650c0 2 all 8 watchers are set, enabling cache 2020-07-17T12:03:07.554 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.547+0000 7f9ee83650c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-17T12:03:07.554 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.547+0000 7f9ee83650c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-17T12:03:07.554 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.547+0000 7f9ea17ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-17T12:03:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.631+0000 7f9e0bfff700 20 reqs_thread_entry: start 2020-07-17T12:03:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.631+0000 7f9e09ffb700 20 reqs_thread_entry: start 2020-07-17T12:03:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.631+0000 7f9ee83650c0 20 init_complete bucket index max shards: 11 2020-07-17T12:03:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.631+0000 7f9dfbfff700 20 reqs_thread_entry: start 2020-07-17T12:03:07.664 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:03:07.664 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.659+0000 7f9ea17ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-17T12:03:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.659+0000 7f9ee83650c0 20 remove_watcher() i=0 2020-07-17T12:03:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.659+0000 7f9ee83650c0 2 removed watcher, disabling cache 2020-07-17T12:03:07.670 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.663+0000 7f9ee83650c0 20 remove_watcher() i=1 2020-07-17T12:03:07.673 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.667+0000 7f9ee83650c0 20 remove_watcher() i=2 2020-07-17T12:03:07.675 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.671+0000 7f9ee83650c0 20 remove_watcher() i=3 2020-07-17T12:03:07.678 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.671+0000 7f9ee83650c0 20 remove_watcher() i=4 2020-07-17T12:03:07.680 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.675+0000 7f9ee83650c0 20 remove_watcher() i=5 2020-07-17T12:03:07.683 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.675+0000 7f9ee83650c0 20 remove_watcher() i=6 2020-07-17T12:03:07.686 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.679+0000 7f9ee83650c0 20 remove_watcher() i=7 2020-07-17T12:03:07.693 DEBUG:tasks.util.rgw: json result: [] 2020-07-17T12:03:07.693 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-17T12:03:07.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-17T12:03:07.694 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-17T12:03:07.779 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:07.780 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:07.791 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.783+0000 7f0edd2860c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-17T12:03:07.791 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.783+0000 7f0edd2860c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-17T12:03:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.791+0000 7f0edd2860c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.791+0000 7f0e0dffb700 20 reqs_thread_entry: start 2020-07-17T12:03:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.791+0000 7f0edd2860c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:07.800 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.791+0000 7f0edd2860c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.801 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.795+0000 7f0edd2860c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-17T12:03:07.801 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.795+0000 7f0edd2860c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.795+0000 7f0edd2860c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.795+0000 7f0edd2860c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.806 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.795+0000 7f0edd2860c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:07.806 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.795+0000 7f0edd2860c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:07.807 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:07.795+0000 7f0edd2860c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:03:07.809 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "efbb56bb-7d12-40f5-a59d-201e34b42071", 2020-07-17T12:03:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default", 2020-07-17T12:03:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "domain_root": "default.rgw.meta:root", 2020-07-17T12:03:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "control_pool": "default.rgw.control", 2020-07-17T12:03:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "gc_pool": "default.rgw.log:gc", 2020-07-17T12:03:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "lc_pool": "default.rgw.log:lc", 2020-07-17T12:03:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "log_pool": "default.rgw.log", 2020-07-17T12:03:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-07-17T12:03:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-07-17T12:03:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "roles_pool": "default.rgw.meta:roles", 2020-07-17T12:03:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-07-17T12:03:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-07-17T12:03:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-07-17T12:03:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-07-17T12:03:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-07-17T12:03:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "otp_pool": "default.rgw.otp", 2020-07-17T12:03:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "system_key": { 2020-07-17T12:03:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "access_key": "", 2020-07-17T12:03:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "secret_key": "" 2020-07-17T12:03:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_pools": [ 2020-07-17T12:03:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "default-placement", 2020-07-17T12:03:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:03:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "index_pool": "default.rgw.buckets.index", 2020-07-17T12:03:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": { 2020-07-17T12:03:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD": { 2020-07-17T12:03:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "data_pool": "default.rgw.buckets.data", 2020-07-17T12:03:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: "compression_type": "random" 2020-07-17T12:03:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-07-17T12:03:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "index_type": 0 2020-07-17T12:03:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "" 2020-07-17T12:03:07.822 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:07.823 DEBUG:tasks.util.rgw: json result: {'id': 'efbb56bb-7d12-40f5-a59d-201e34b42071', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-17T12:03:07.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-07-17T12:03:07.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-07-17T12:03:07.824 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-07-17T12:03:07.859 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:07.859 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:07.887 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:07.863+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.878 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.871+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.878 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.871+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.882 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.875+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.882 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.875+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.883 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.875+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.883 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.875+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.883 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.879+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.904 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.904 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.916 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.916 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 10 cannot find current period zonegroup using local zonegroup 2020-07-17T12:03:09.916 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.917 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.917 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.918 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.918 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 10 Creating default zonegroup 2020-07-17T12:03:09.918 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 20 get_system_obj_state: rctx=0x7ffead1d55b0 obj=:.pools.avail state=0x563eb4130f40 s->prefetch_data=0 2020-07-17T12:03:09.919 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.899+0000 7f58599d50c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-07-17T12:03:09.923 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.919+0000 7f58599d50c0 10 WARNING: store->list_zones() returned r=-2 2020-07-17T12:03:09.923 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.919+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.923 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.919+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.928 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.923+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.929 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.923+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.929 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.923+0000 7f58599d50c0 10 could not read realm id: (2) No such file or directory 2020-07-17T12:03:09.929 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.923+0000 7f58599d50c0 10 WARNING: failed to set zone as default, r=-22 2020-07-17T12:03:09.930 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.923+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.930 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.923+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.936 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.931+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.937 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.931+0000 7f58599d50c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-17T12:03:09.937 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.931+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.938 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.931+0000 7f58599d50c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:03:09.938 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.931+0000 7f58599d50c0 20 zonegroup default 2020-07-17T12:03:09.938 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.931+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.939 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.935+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:09.939 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.935+0000 7f58599d50c0 10 Cannot find current period zone using local zone 2020-07-17T12:03:09.940 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.935+0000 7f58599d50c0 10 Using default name default 2020-07-17T12:03:09.940 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.935+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.940 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.935+0000 7f58599d50c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:09.941 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.935+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:09.941 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.935+0000 7f58599d50c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-17T12:03:09.941 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.935+0000 7f58599d50c0 20 zone default found 2020-07-17T12:03:09.942 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:09.935+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:11.899 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:11.895+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:11.899 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:11.895+0000 7f58599d50c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:11.900 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:11.895+0000 7f58599d50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:11.901 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:11.895+0000 7f58599d50c0 20 started sync module instance, tier type = 2020-07-17T12:03:11.901 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:11.895+0000 7f58599d50c0 20 started zone id=504f036e-2f66-4dba-ae08-a898b9a4fd83 (name=default) with tier type = 2020-07-17T12:03:13.939 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 add_watcher() i=0 2020-07-17T12:03:13.940 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 add_watcher() i=1 2020-07-17T12:03:13.940 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 add_watcher() i=2 2020-07-17T12:03:13.941 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 add_watcher() i=3 2020-07-17T12:03:13.941 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 add_watcher() i=4 2020-07-17T12:03:13.941 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 add_watcher() i=5 2020-07-17T12:03:13.942 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 add_watcher() i=6 2020-07-17T12:03:13.943 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 add_watcher() i=7 2020-07-17T12:03:13.943 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 2 all 8 watchers are set, enabling cache 2020-07-17T12:03:13.943 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-17T12:03:13.944 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f58599d50c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-17T12:03:13.944 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:13.935+0000 7f5812fe5700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-17T12:03:17.921 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:17.914+0000 7f58599d50c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-07-17T12:03:18.026 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.018+0000 7f5781ffb700 20 reqs_thread_entry: start 2020-07-17T12:03:18.026 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.018+0000 7f58599d50c0 20 init_complete bucket index max shards: 11 2020-07-17T12:03:18.027 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.018+0000 7f576b7fe700 20 reqs_thread_entry: start 2020-07-17T12:03:18.027 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.018+0000 7f5769ffb700 20 reqs_thread_entry: start 2020-07-17T12:03:18.051 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2020-07-17T12:03:18.051 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.042+0000 7f5812fe5700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-17T12:03:18.054 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.046+0000 7f58599d50c0 20 remove_watcher() i=0 2020-07-17T12:03:18.054 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.046+0000 7f58599d50c0 2 removed watcher, disabling cache 2020-07-17T12:03:18.057 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.050+0000 7f58599d50c0 20 remove_watcher() i=1 2020-07-17T12:03:18.061 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.054+0000 7f58599d50c0 20 remove_watcher() i=2 2020-07-17T12:03:18.064 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.058+0000 7f58599d50c0 20 remove_watcher() i=3 2020-07-17T12:03:18.067 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.058+0000 7f58599d50c0 20 remove_watcher() i=4 2020-07-17T12:03:18.070 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.062+0000 7f58599d50c0 20 remove_watcher() i=5 2020-07-17T12:03:18.073 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.066+0000 7f58599d50c0 20 remove_watcher() i=6 2020-07-17T12:03:18.076 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.070+0000 7f58599d50c0 20 remove_watcher() i=7 2020-07-17T12:03:18.085 DEBUG:tasks.util.rgw: json result: [] 2020-07-17T12:03:18.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-17T12:03:18.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-17T12:03:18.087 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-17T12:03:18.166 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:18.166 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:18.180 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.174+0000 7f9ab3d960c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-17T12:03:18.180 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.174+0000 7f9ab3d960c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-17T12:03:18.186 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.178+0000 7f9a24ff9700 20 reqs_thread_entry: start 2020-07-17T12:03:18.187 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.178+0000 7f9ab3d960c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.187 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.182+0000 7f9ab3d960c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:18.188 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.182+0000 7f9ab3d960c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.188 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.182+0000 7f9ab3d960c0 20 rados_obj.operate() r=0 bl.length=839 2020-07-17T12:03:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.182+0000 7f9ab3d960c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.182+0000 7f9ab3d960c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.182+0000 7f9ab3d960c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.190 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.182+0000 7f9ab3d960c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:18.190 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.182+0000 7f9ab3d960c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.191 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.182+0000 7f9ab3d960c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:03:18.198 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:03:18.198 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "504f036e-2f66-4dba-ae08-a898b9a4fd83", 2020-07-17T12:03:18.199 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default", 2020-07-17T12:03:18.199 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "default.rgw.meta:root", 2020-07-17T12:03:18.199 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "default.rgw.control", 2020-07-17T12:03:18.199 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "default.rgw.log:gc", 2020-07-17T12:03:18.200 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "default.rgw.log:lc", 2020-07-17T12:03:18.200 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "default.rgw.log", 2020-07-17T12:03:18.200 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-07-17T12:03:18.200 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-07-17T12:03:18.200 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "default.rgw.meta:roles", 2020-07-17T12:03:18.200 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-07-17T12:03:18.201 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-07-17T12:03:18.201 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-07-17T12:03:18.201 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-07-17T12:03:18.201 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-07-17T12:03:18.201 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "default.rgw.otp", 2020-07-17T12:03:18.202 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2020-07-17T12:03:18.202 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "", 2020-07-17T12:03:18.202 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "" 2020-07-17T12:03:18.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:18.202 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2020-07-17T12:03:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2020-07-17T12:03:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:03:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "default.rgw.buckets.index", 2020-07-17T12:03:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2020-07-17T12:03:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2020-07-17T12:03:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "default.rgw.buckets.data", 2020-07-17T12:03:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2020-07-17T12:03:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-07-17T12:03:18.205 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2020-07-17T12:03:18.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:18.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:18.205 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:18.205 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "" 2020-07-17T12:03:18.205 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:03:18.206 DEBUG:tasks.util.rgw: json result: {'id': '504f036e-2f66-4dba-ae08-a898b9a4fd83', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-17T12:03:18.209 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-07-17T12:03:18.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-07-17T12:03:18.210 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-07-17T12:03:18.280 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:18.281 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:18.294 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.286+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.296 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.290+0000 7fbd3c3a00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.296 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.290+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.297 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.290+0000 7fbd3c3a00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.298 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.290+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.298 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.290+0000 7fbd3c3a00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.299 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.290+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.299 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.290+0000 7fbd3c3a00c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 RGWRados::pool_iterate: got zonegroup_info.12c9dac4-ab9e-4e4e-ad73-32a16025a04a 2020-07-17T12:03:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 RGWRados::pool_iterate: got zone_info.504f036e-2f66-4dba-ae08-a898b9a4fd83 2020-07-17T12:03:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 RGWRados::pool_iterate: got zone_names.default 2020-07-17T12:03:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-07-17T12:03:18.326 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.326 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:18.327 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.327 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-17T12:03:18.327 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.328 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:18.329 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.318+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.329 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:03:18.329 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.330 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.330 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 10 cannot find current period zonegroup using local zonegroup 2020-07-17T12:03:18.330 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.332 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.333 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.333 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:18.333 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.334 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:03:18.334 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 zonegroup default 2020-07-17T12:03:18.335 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.322+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.335 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.335 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 10 Cannot find current period zone using local zone 2020-07-17T12:03:18.336 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.336 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.336 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.338 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:18.338 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.339 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-17T12:03:18.339 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 zone default found 2020-07-17T12:03:18.339 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.340 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.340 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.326+0000 7fbd3c3a00c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.340 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.330+0000 7fbd3c3a00c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.341 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.330+0000 7fbd3c3a00c0 20 started sync module instance, tier type = 2020-07-17T12:03:18.341 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.330+0000 7fbd3c3a00c0 20 started zone id=504f036e-2f66-4dba-ae08-a898b9a4fd83 (name=default) with tier type = 2020-07-17T12:03:18.358 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.350+0000 7fbd3c3a00c0 20 add_watcher() i=0 2020-07-17T12:03:18.359 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.350+0000 7fbd3c3a00c0 20 add_watcher() i=1 2020-07-17T12:03:18.359 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.350+0000 7fbd3c3a00c0 20 add_watcher() i=2 2020-07-17T12:03:18.359 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.350+0000 7fbd3c3a00c0 20 add_watcher() i=3 2020-07-17T12:03:18.359 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.350+0000 7fbd3c3a00c0 20 add_watcher() i=4 2020-07-17T12:03:18.359 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.350+0000 7fbd3c3a00c0 20 add_watcher() i=5 2020-07-17T12:03:18.360 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.350+0000 7fbd3c3a00c0 20 add_watcher() i=6 2020-07-17T12:03:18.360 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.354+0000 7fbd3c3a00c0 20 add_watcher() i=7 2020-07-17T12:03:18.361 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.354+0000 7fbd3c3a00c0 2 all 8 watchers are set, enabling cache 2020-07-17T12:03:18.361 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.354+0000 7fbd3c3a00c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-17T12:03:18.361 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.354+0000 7fbd3c3a00c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-17T12:03:18.361 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.354+0000 7fbcf57ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-17T12:03:18.454 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.446+0000 7fbc5ffff700 20 reqs_thread_entry: start 2020-07-17T12:03:18.454 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.446+0000 7fbc5dffb700 20 reqs_thread_entry: start 2020-07-17T12:03:18.454 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.446+0000 7fbd3c3a00c0 20 init_complete bucket index max shards: 11 2020-07-17T12:03:18.455 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.446+0000 7fbc4ffff700 20 reqs_thread_entry: start 2020-07-17T12:03:18.482 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2020-07-17T12:03:18.482 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.474+0000 7fbcf57ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-17T12:03:18.484 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.478+0000 7fbd3c3a00c0 20 remove_watcher() i=0 2020-07-17T12:03:18.484 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.478+0000 7fbd3c3a00c0 2 removed watcher, disabling cache 2020-07-17T12:03:18.487 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.478+0000 7fbd3c3a00c0 20 remove_watcher() i=1 2020-07-17T12:03:18.490 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.482+0000 7fbd3c3a00c0 20 remove_watcher() i=2 2020-07-17T12:03:18.493 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.486+0000 7fbd3c3a00c0 20 remove_watcher() i=3 2020-07-17T12:03:18.496 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.490+0000 7fbd3c3a00c0 20 remove_watcher() i=4 2020-07-17T12:03:18.499 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.494+0000 7fbd3c3a00c0 20 remove_watcher() i=5 2020-07-17T12:03:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.494+0000 7fbd3c3a00c0 20 remove_watcher() i=6 2020-07-17T12:03:18.505 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.498+0000 7fbd3c3a00c0 20 remove_watcher() i=7 2020-07-17T12:03:18.513 DEBUG:tasks.util.rgw: json result: [] 2020-07-17T12:03:18.513 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-17T12:03:18.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-17T12:03:18.514 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-17T12:03:18.599 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:18.599 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:18.610 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.602+0000 7fd42fd3a0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-17T12:03:18.610 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.602+0000 7fd42fd3a0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-17T12:03:18.617 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd42fd3a0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.617 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd3d0ff9700 20 reqs_thread_entry: start 2020-07-17T12:03:18.618 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd42fd3a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:18.619 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd42fd3a0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.619 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd42fd3a0c0 20 rados_obj.operate() r=0 bl.length=855 2020-07-17T12:03:18.619 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd42fd3a0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.621 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd42fd3a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-07-17T12:03:18.621 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd42fd3a0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.621 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd42fd3a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-07-17T12:03:18.622 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.610+0000 7fd42fd3a0c0 20 rados->read ofs=0 len=0 2020-07-17T12:03:18.622 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:18.614+0000 7fd42fd3a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-07-17T12:03:18.627 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:03:18.628 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "504f036e-2f66-4dba-ae08-a898b9a4fd83", 2020-07-17T12:03:18.628 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default", 2020-07-17T12:03:18.629 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "default.rgw.meta:root", 2020-07-17T12:03:18.629 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "default.rgw.control", 2020-07-17T12:03:18.629 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "default.rgw.log:gc", 2020-07-17T12:03:18.630 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "default.rgw.log:lc", 2020-07-17T12:03:18.630 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "default.rgw.log", 2020-07-17T12:03:18.630 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-07-17T12:03:18.631 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-07-17T12:03:18.631 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "default.rgw.meta:roles", 2020-07-17T12:03:18.631 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-07-17T12:03:18.632 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-07-17T12:03:18.632 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-07-17T12:03:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-07-17T12:03:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-07-17T12:03:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "default.rgw.otp", 2020-07-17T12:03:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2020-07-17T12:03:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "", 2020-07-17T12:03:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "" 2020-07-17T12:03:18.635 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:18.635 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2020-07-17T12:03:18.636 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:18.636 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2020-07-17T12:03:18.636 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:03:18.637 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "default.rgw.buckets.index", 2020-07-17T12:03:18.637 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2020-07-17T12:03:18.638 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2020-07-17T12:03:18.639 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "default.rgw.buckets.data", 2020-07-17T12:03:18.639 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2020-07-17T12:03:18.640 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:18.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:18.640 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-07-17T12:03:18.641 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2020-07-17T12:03:18.641 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:18.641 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:18.642 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:18.642 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "" 2020-07-17T12:03:18.642 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:03:18.644 DEBUG:tasks.util.rgw: json result: {'id': '504f036e-2f66-4dba-ae08-a898b9a4fd83', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-17T12:03:18.644 INFO:tasks.rgw:Starting rgw... 2020-07-17T12:03:18.645 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-07-17T12:03:18.646 INFO:tasks.rgw:Using beast as radosgw frontend 2020-07-17T12:03:18.646 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-07-17T12:03:18.648 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-07-17T12:03:18.654 INFO:tasks.rgw.c1.client.0:Started 2020-07-17T12:03:18.655 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-07-17T12:03:18.655 INFO:tasks.rgw:Using beast as radosgw frontend 2020-07-17T12:03:18.656 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-07-17T12:03:18.657 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-07-17T12:03:18.660 INFO:tasks.rgw.c1.client.1:Started 2020-07-17T12:03:18.661 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-07-17T12:03:18.662 INFO:tasks.rgw:Using beast as radosgw frontend 2020-07-17T12:03:18.662 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-07-17T12:03:18.663 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-07-17T12:03:18.689 INFO:tasks.rgw.c2.client.0:Started 2020-07-17T12:03:18.690 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-07-17T12:03:18.691 INFO:tasks.rgw:Using beast as radosgw frontend 2020-07-17T12:03:18.691 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-07-17T12:03:18.692 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-07-17T12:03:18.697 INFO:tasks.rgw.c2.client.1:Started 2020-07-17T12:03:18.698 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi146.front.sepia.ceph.com:8000/ 2020-07-17T12:03:18.698 INFO:teuthology.orchestra.run.smithi146:> curl http://smithi146.front.sepia.ceph.com:8000/ 2020-07-17T12:03:18.715 INFO:teuthology.orchestra.run.smithi146.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:03:18.715 INFO:teuthology.orchestra.run.smithi146.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:03:18.718 INFO:teuthology.orchestra.run.smithi146.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi146.front.sepia.ceph.com port 8000: Connection refused 2020-07-17T12:03:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-17T12:03:19.720 INFO:teuthology.orchestra.run.smithi146:> curl http://smithi146.front.sepia.ceph.com:8000/ 2020-07-17T12:03:19.735 INFO:teuthology.orchestra.run.smithi146.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:03:19.735 INFO:teuthology.orchestra.run.smithi146.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:03:19.739 INFO:teuthology.orchestra.run.smithi146.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-17T12:03:19.740 INFO:teuthology.orchestra.run.smithi146.stdout:anonymous 2020-07-17T12:03:19.740 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi146.front.sepia.ceph.com:8001/ 2020-07-17T12:03:19.741 INFO:teuthology.orchestra.run.smithi146:> curl http://smithi146.front.sepia.ceph.com:8001/ 2020-07-17T12:03:19.794 INFO:teuthology.orchestra.run.smithi146.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:03:19.794 INFO:teuthology.orchestra.run.smithi146.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:03:19.799 INFO:teuthology.orchestra.run.smithi146.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-17T12:03:19.799 INFO:teuthology.orchestra.run.smithi146.stdout:anonymous 2020-07-17T12:03:19.800 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi052.front.sepia.ceph.com:8000/ 2020-07-17T12:03:19.800 INFO:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000/ 2020-07-17T12:03:19.816 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:03:19.817 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:03:19.822 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-17T12:03:19.822 INFO:teuthology.orchestra.run.smithi052.stdout:anonymous 2020-07-17T12:03:19.823 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi052.front.sepia.ceph.com:8001/ 2020-07-17T12:03:19.824 INFO:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8001/ 2020-07-17T12:03:19.878 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:03:19.879 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:03:19.884 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-17T12:03:19.884 INFO:teuthology.orchestra.run.smithi052.stdout:anonymous 2020-07-17T12:03:19.885 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-07-17T12:03:20.578 INFO:tasks.rgw_multisite:creating realm.. 2020-07-17T12:03:20.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:20.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:20.580 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:20.617 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:20.618 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:20.667 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:20.667 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:03:20.668 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-realm", 2020-07-17T12:03:20.668 INFO:teuthology.orchestra.run.smithi146.stdout: "current_period": "68df446e-5a13-4da6-9e83-5e687570550c", 2020-07-17T12:03:20.668 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 1 2020-07-17T12:03:20.668 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:20.670 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-07-17T12:03:20.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:20.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:20.672 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:20.751 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:20.752 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:20.775 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:20.766+0000 7f5481a7a0c0 -1 old format 2020-07-17T12:03:20.789 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:20.790 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "test-zonegroup", 2020-07-17T12:03:20.790 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zonegroup", 2020-07-17T12:03:20.790 INFO:teuthology.orchestra.run.smithi146.stdout: "api_name": "", 2020-07-17T12:03:20.791 INFO:teuthology.orchestra.run.smithi146.stdout: "is_master": "true", 2020-07-17T12:03:20.791 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:03:20.791 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:03:20.792 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:20.792 INFO:teuthology.orchestra.run.smithi146.stdout: "hostnames": [], 2020-07-17T12:03:20.792 INFO:teuthology.orchestra.run.smithi146.stdout: "hostnames_s3website": [], 2020-07-17T12:03:20.793 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zone": "", 2020-07-17T12:03:20.793 INFO:teuthology.orchestra.run.smithi146.stdout: "zones": [], 2020-07-17T12:03:20.793 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_targets": [], 2020-07-17T12:03:20.794 INFO:teuthology.orchestra.run.smithi146.stdout: "default_placement": "", 2020-07-17T12:03:20.794 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:03:20.794 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_policy": { 2020-07-17T12:03:20.795 INFO:teuthology.orchestra.run.smithi146.stdout: "groups": [] 2020-07-17T12:03:20.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:20.795 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:20.797 INFO:tasks.rgw_multisite:creating master zone.. 2020-07-17T12:03:20.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--endpoints', 'http://smithi146.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:20.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--endpoints', 'http://smithi146.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:20.798 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 1R9V3H5FL4JONCEX --secret LcR3LFWGav9hRFziuKCRCpXMbs1dgJln --endpoints http://smithi146.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:20.886 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:20.886 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:20.955 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:20.955 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:20.955 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone1", 2020-07-17T12:03:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-07-17T12:03:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: "control_pool": "test-zone1.rgw.control", 2020-07-17T12:03:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-07-17T12:03:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-07-17T12:03:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: "log_pool": "test-zone1.rgw.log", 2020-07-17T12:03:20.957 INFO:teuthology.orchestra.run.smithi146.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-07-17T12:03:20.957 INFO:teuthology.orchestra.run.smithi146.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-07-17T12:03:20.957 INFO:teuthology.orchestra.run.smithi146.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-07-17T12:03:20.957 INFO:teuthology.orchestra.run.smithi146.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-07-17T12:03:20.957 INFO:teuthology.orchestra.run.smithi146.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-07-17T12:03:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-07-17T12:03:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-07-17T12:03:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-07-17T12:03:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-07-17T12:03:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: "system_key": { 2020-07-17T12:03:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:03:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:03:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_pools": [ 2020-07-17T12:03:20.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:20.960 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "default-placement", 2020-07-17T12:03:20.960 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:03:20.960 INFO:teuthology.orchestra.run.smithi146.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-07-17T12:03:20.960 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": { 2020-07-17T12:03:20.960 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD": { 2020-07-17T12:03:20.961 INFO:teuthology.orchestra.run.smithi146.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2020-07-17T12:03:20.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:20.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:20.961 INFO:teuthology.orchestra.run.smithi146.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-07-17T12:03:20.962 INFO:teuthology.orchestra.run.smithi146.stdout: "index_type": 0 2020-07-17T12:03:20.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:20.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:20.962 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:20.962 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca" 2020-07-17T12:03:20.963 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:20.964 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-07-17T12:03:22.178 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-07-17T12:03:22.193 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-07-17T12:03:23.186 INFO:teuthology.orchestra.run.smithi146.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-07-17T12:03:23.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:23.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:23.216 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:23.255 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:23.255 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:23.278 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:23.278 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:23.279 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone1", 2020-07-17T12:03:23.279 INFO:teuthology.orchestra.run.smithi146.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-07-17T12:03:23.279 INFO:teuthology.orchestra.run.smithi146.stdout: "control_pool": "test-zone1.rgw.control", 2020-07-17T12:03:23.279 INFO:teuthology.orchestra.run.smithi146.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-07-17T12:03:23.279 INFO:teuthology.orchestra.run.smithi146.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-07-17T12:03:23.280 INFO:teuthology.orchestra.run.smithi146.stdout: "log_pool": "test-zone1.rgw.log", 2020-07-17T12:03:23.280 INFO:teuthology.orchestra.run.smithi146.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-07-17T12:03:23.280 INFO:teuthology.orchestra.run.smithi146.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-07-17T12:03:23.280 INFO:teuthology.orchestra.run.smithi146.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-07-17T12:03:23.280 INFO:teuthology.orchestra.run.smithi146.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-07-17T12:03:23.281 INFO:teuthology.orchestra.run.smithi146.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-07-17T12:03:23.281 INFO:teuthology.orchestra.run.smithi146.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-07-17T12:03:23.281 INFO:teuthology.orchestra.run.smithi146.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-07-17T12:03:23.281 INFO:teuthology.orchestra.run.smithi146.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-07-17T12:03:23.281 INFO:teuthology.orchestra.run.smithi146.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-07-17T12:03:23.282 INFO:teuthology.orchestra.run.smithi146.stdout: "system_key": { 2020-07-17T12:03:23.282 INFO:teuthology.orchestra.run.smithi146.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:03:23.282 INFO:teuthology.orchestra.run.smithi146.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:03:23.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:23.282 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_pools": [ 2020-07-17T12:03:23.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:23.283 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "default-placement", 2020-07-17T12:03:23.283 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:03:23.283 INFO:teuthology.orchestra.run.smithi146.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-07-17T12:03:23.284 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": { 2020-07-17T12:03:23.284 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD": { 2020-07-17T12:03:23.284 INFO:teuthology.orchestra.run.smithi146.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-07-17T12:03:23.284 INFO:teuthology.orchestra.run.smithi146.stdout: "compression_type": "random" 2020-07-17T12:03:23.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:23.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:23.285 INFO:teuthology.orchestra.run.smithi146.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-07-17T12:03:23.285 INFO:teuthology.orchestra.run.smithi146.stdout: "index_type": 0 2020-07-17T12:03:23.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:23.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:23.285 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:23.286 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "" 2020-07-17T12:03:23.286 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:23.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:23.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:23.288 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:23.364 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:23.365 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:31.439 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:31.439 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:03:31.440 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 1, 2020-07-17T12:03:31.440 INFO:teuthology.orchestra.run.smithi146.stdout: "predecessor_uuid": "68df446e-5a13-4da6-9e83-5e687570550c", 2020-07-17T12:03:31.440 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": [], 2020-07-17T12:03:31.440 INFO:teuthology.orchestra.run.smithi146.stdout: "period_map": { 2020-07-17T12:03:31.440 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:03:31.441 INFO:teuthology.orchestra.run.smithi146.stdout: "zonegroups": [ 2020-07-17T12:03:31.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:31.441 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "test-zonegroup", 2020-07-17T12:03:31.441 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zonegroup", 2020-07-17T12:03:31.442 INFO:teuthology.orchestra.run.smithi146.stdout: "api_name": "", 2020-07-17T12:03:31.442 INFO:teuthology.orchestra.run.smithi146.stdout: "is_master": "true", 2020-07-17T12:03:31.442 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:03:31.442 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:03:31.442 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:31.442 INFO:teuthology.orchestra.run.smithi146.stdout: "hostnames": [], 2020-07-17T12:03:31.443 INFO:teuthology.orchestra.run.smithi146.stdout: "hostnames_s3website": [], 2020-07-17T12:03:31.443 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:31.443 INFO:teuthology.orchestra.run.smithi146.stdout: "zones": [ 2020-07-17T12:03:31.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:31.443 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:31.444 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone1", 2020-07-17T12:03:31.444 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:03:31.444 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:03:31.444 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:31.444 INFO:teuthology.orchestra.run.smithi146.stdout: "log_meta": "false", 2020-07-17T12:03:31.444 INFO:teuthology.orchestra.run.smithi146.stdout: "log_data": "false", 2020-07-17T12:03:31.445 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:03:31.445 INFO:teuthology.orchestra.run.smithi146.stdout: "read_only": "false", 2020-07-17T12:03:31.445 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_type": "", 2020-07-17T12:03:31.445 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from_all": "true", 2020-07-17T12:03:31.445 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from": [], 2020-07-17T12:03:31.446 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_zone": "" 2020-07-17T12:03:31.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:31.446 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:31.446 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_targets": [ 2020-07-17T12:03:31.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:31.446 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default-placement", 2020-07-17T12:03:31.447 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2020-07-17T12:03:31.447 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": [ 2020-07-17T12:03:31.447 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD" 2020-07-17T12:03:31.447 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:03:31.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:31.448 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:31.448 INFO:teuthology.orchestra.run.smithi146.stdout: "default_placement": "default-placement", 2020-07-17T12:03:31.448 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:03:31.448 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_policy": { 2020-07-17T12:03:31.448 INFO:teuthology.orchestra.run.smithi146.stdout: "groups": [] 2020-07-17T12:03:31.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:31.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:31.449 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:31.449 INFO:teuthology.orchestra.run.smithi146.stdout: "short_zone_ids": [ 2020-07-17T12:03:31.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:31.449 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:31.450 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 340514820 2020-07-17T12:03:31.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:31.451 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:03:31.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:31.451 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zonegroup": "test-zonegroup", 2020-07-17T12:03:31.451 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:31.452 INFO:teuthology.orchestra.run.smithi146.stdout: "period_config": { 2020-07-17T12:03:31.452 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_quota": { 2020-07-17T12:03:31.452 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:03:31.452 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:03:31.452 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:03:31.452 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:03:31.453 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:03:31.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:31.453 INFO:teuthology.orchestra.run.smithi146.stdout: "user_quota": { 2020-07-17T12:03:31.453 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:03:31.453 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:03:31.454 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:03:31.454 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:03:31.454 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:03:31.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:31.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:31.454 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:03:31.455 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_name": "test-realm", 2020-07-17T12:03:31.455 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:03:31.455 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:31.456 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-07-17T12:03:31.457 INFO:teuthology.orchestra.run:waiting for 300 2020-07-17T12:03:31.501 INFO:tasks.rgw.c1.client.0.smithi146.stdout:2020-07-17T12:03:31.494+0000 7f7b09b0f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 13517) UID: 0 2020-07-17T12:03:31.501 INFO:tasks.rgw.c1.client.0.smithi146.stdout:2020-07-17T12:03:31.494+0000 7f7b211e3100 -1 shutting down 2020-07-17T12:03:31.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-07-17T12:03:37.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2020-07-17T12:03:37.559 INFO:tasks.rgw.c1.client.0:Stopped 2020-07-17T12:03:37.560 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-07-17T12:03:37.560 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-07-17T12:03:37.574 INFO:tasks.rgw.c1.client.0:Started 2020-07-17T12:03:37.575 INFO:teuthology.orchestra.run.smithi146:> curl http://smithi146.front.sepia.ceph.com:8000 2020-07-17T12:03:37.586 INFO:teuthology.orchestra.run.smithi146.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:03:37.587 INFO:teuthology.orchestra.run.smithi146.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:03:37.591 INFO:teuthology.orchestra.run.smithi146.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi146.front.sepia.ceph.com port 8000: Connection refused 2020-07-17T12:03:37.592 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-17T12:03:38.593 INFO:teuthology.orchestra.run.smithi146:> curl http://smithi146.front.sepia.ceph.com:8000 2020-07-17T12:03:38.610 INFO:teuthology.orchestra.run.smithi146.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:03:38.611 INFO:teuthology.orchestra.run.smithi146.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:03:38.615 INFO:teuthology.orchestra.run.smithi146.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-17T12:03:38.616 INFO:teuthology.orchestra.run.smithi146.stdout:anonymous 2020-07-17T12:03:38.617 INFO:tasks.rgw_multisite:creating admin user.. 2020-07-17T12:03:38.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:38.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:38.618 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 1R9V3H5FL4JONCEX --secret LcR3LFWGav9hRFziuKCRCpXMbs1dgJln --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:38.707 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:38.707 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:38.902 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:38.902 INFO:teuthology.orchestra.run.smithi146.stdout: "user_id": "realm-admin", 2020-07-17T12:03:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "display_name": "Realm Admin", 2020-07-17T12:03:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "email": "", 2020-07-17T12:03:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "suspended": 0, 2020-07-17T12:03:38.903 INFO:teuthology.orchestra.run.smithi146.stdout: "max_buckets": 1000, 2020-07-17T12:03:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "subusers": [], 2020-07-17T12:03:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: "keys": [ 2020-07-17T12:03:38.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "user": "realm-admin", 2020-07-17T12:03:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:03:38.905 INFO:teuthology.orchestra.run.smithi146.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:03:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:38.906 INFO:teuthology.orchestra.run.smithi146.stdout: "swift_keys": [], 2020-07-17T12:03:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": [], 2020-07-17T12:03:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "op_mask": "read, write, delete", 2020-07-17T12:03:38.907 INFO:teuthology.orchestra.run.smithi146.stdout: "system": "true", 2020-07-17T12:03:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: "default_placement": "", 2020-07-17T12:03:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: "default_storage_class": "", 2020-07-17T12:03:38.908 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_tags": [], 2020-07-17T12:03:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_quota": { 2020-07-17T12:03:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:03:38.909 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:03:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:03:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:03:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:03:38.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: "user_quota": { 2020-07-17T12:03:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:03:38.911 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:03:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:03:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:03:38.912 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:03:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: "temp_url_keys": [], 2020-07-17T12:03:38.913 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "rgw", 2020-07-17T12:03:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: "mfa_ids": [] 2020-07-17T12:03:38.919 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:03:38.929 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-07-17T12:03:38.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi146.front.sepia.ceph.com:8000', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:38.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi146.front.sepia.ceph.com:8000', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:38.931 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi146.front.sepia.ceph.com:8000 --access-key 1R9V3H5FL4JONCEX --secret LcR3LFWGav9hRFziuKCRCpXMbs1dgJln --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:38.975 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:39.095 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:03:39.095 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:03:39.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-realm", 2020-07-17T12:03:39.095 INFO:teuthology.orchestra.run.smithi052.stdout: "current_period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:03:39.096 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 2 2020-07-17T12:03:39.096 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:03:39.097 INFO:tasks.rgw_multisite:creating zone.. 2020-07-17T12:03:39.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:39.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:39.098 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 1R9V3H5FL4JONCEX --secret LcR3LFWGav9hRFziuKCRCpXMbs1dgJln --endpoints http://smithi052.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:39.185 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:39.185 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:39.251 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:39.242+0000 7fb51a2c20c0 0 failed reading obj info from .rgw.root:zone_info.bc359d1d-af48-401e-ac43-335f84c3f1d9: (2) No such file or directory 2020-07-17T12:03:39.252 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:03:39.242+0000 7fb51a2c20c0 0 WARNING: could not read zone params for zone id=bc359d1d-af48-401e-ac43-335f84c3f1d9 name=test-zone1 2020-07-17T12:03:39.263 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:03:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:03:39.264 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2020-07-17T12:03:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-07-17T12:03:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone2.rgw.control", 2020-07-17T12:03:39.265 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-07-17T12:03:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-07-17T12:03:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone2.rgw.log", 2020-07-17T12:03:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-07-17T12:03:39.266 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-07-17T12:03:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-07-17T12:03:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-07-17T12:03:39.267 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-07-17T12:03:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-07-17T12:03:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-07-17T12:03:39.268 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-07-17T12:03:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-07-17T12:03:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2020-07-17T12:03:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:03:39.269 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:03:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2020-07-17T12:03:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:39.270 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2020-07-17T12:03:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:03:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-07-17T12:03:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2020-07-17T12:03:39.271 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2020-07-17T12:03:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2020-07-17T12:03:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:39.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-07-17T12:03:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2020-07-17T12:03:39.273 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:39.274 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca" 2020-07-17T12:03:39.275 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:03:39.276 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-07-17T12:03:40.027 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-07-17T12:03:40.041 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-07-17T12:03:41.022 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-07-17T12:03:41.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:41.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:41.039 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:41.084 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:41.108 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:03:41.109 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:03:41.109 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2020-07-17T12:03:41.110 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-07-17T12:03:41.110 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone2.rgw.control", 2020-07-17T12:03:41.110 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-07-17T12:03:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-07-17T12:03:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone2.rgw.log", 2020-07-17T12:03:41.111 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-07-17T12:03:41.112 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-07-17T12:03:41.112 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-07-17T12:03:41.112 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-07-17T12:03:41.113 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-07-17T12:03:41.113 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-07-17T12:03:41.113 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-07-17T12:03:41.113 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-07-17T12:03:41.114 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-07-17T12:03:41.114 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2020-07-17T12:03:41.114 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:03:41.115 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:03:41.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:41.115 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2020-07-17T12:03:41.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:41.116 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2020-07-17T12:03:41.116 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:03:41.117 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-07-17T12:03:41.117 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2020-07-17T12:03:41.117 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2020-07-17T12:03:41.118 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-07-17T12:03:41.118 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2020-07-17T12:03:41.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:41.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:41.119 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-07-17T12:03:41.119 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2020-07-17T12:03:41.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:41.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:41.121 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:41.121 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "" 2020-07-17T12:03:41.121 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:03:41.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:41.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:41.124 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:41.165 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:41.165 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:43.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-07-17T12:03:49.200 INFO:teuthology.orchestra.run.smithi052.stderr:Sending period to new master zone bc359d1d-af48-401e-ac43-335f84c3f1d9 2020-07-17T12:03:49.286 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:03:49.287 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:03:49.287 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 2, 2020-07-17T12:03:49.287 INFO:teuthology.orchestra.run.smithi052.stdout: "predecessor_uuid": "68df446e-5a13-4da6-9e83-5e687570550c", 2020-07-17T12:03:49.288 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": [], 2020-07-17T12:03:49.288 INFO:teuthology.orchestra.run.smithi052.stdout: "period_map": { 2020-07-17T12:03:49.288 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:03:49.289 INFO:teuthology.orchestra.run.smithi052.stdout: "zonegroups": [ 2020-07-17T12:03:49.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:49.289 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "test-zonegroup", 2020-07-17T12:03:49.290 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zonegroup", 2020-07-17T12:03:49.290 INFO:teuthology.orchestra.run.smithi052.stdout: "api_name": "", 2020-07-17T12:03:49.290 INFO:teuthology.orchestra.run.smithi052.stdout: "is_master": "true", 2020-07-17T12:03:49.291 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:03:49.291 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:03:49.291 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:49.292 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames": [], 2020-07-17T12:03:49.292 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames_s3website": [], 2020-07-17T12:03:49.292 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:49.293 INFO:teuthology.orchestra.run.smithi052.stdout: "zones": [ 2020-07-17T12:03:49.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:49.294 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:49.294 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone1", 2020-07-17T12:03:49.295 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:03:49.295 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:03:49.295 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:49.296 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:03:49.296 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:03:49.296 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:03:49.297 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:03:49.297 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:03:49.297 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:03:49.298 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:03:49.298 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:03:49.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:49.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:49.299 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:03:49.299 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2020-07-17T12:03:49.300 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:03:49.301 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2020-07-17T12:03:49.301 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:49.302 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:03:49.302 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:03:49.302 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:03:49.303 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:03:49.303 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:03:49.303 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:03:49.304 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:03:49.304 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:03:49.304 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:49.305 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:49.305 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_targets": [ 2020-07-17T12:03:49.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:49.306 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default-placement", 2020-07-17T12:03:49.306 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2020-07-17T12:03:49.306 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": [ 2020-07-17T12:03:49.307 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD" 2020-07-17T12:03:49.307 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:03:49.308 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:49.308 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:49.308 INFO:teuthology.orchestra.run.smithi052.stdout: "default_placement": "default-placement", 2020-07-17T12:03:49.309 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:03:49.309 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_policy": { 2020-07-17T12:03:49.309 INFO:teuthology.orchestra.run.smithi052.stdout: "groups": [] 2020-07-17T12:03:49.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:49.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:49.310 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:03:49.311 INFO:teuthology.orchestra.run.smithi052.stdout: "short_zone_ids": [ 2020-07-17T12:03:49.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:49.311 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:49.312 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 340514820 2020-07-17T12:03:49.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:49.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:03:49.313 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:03:49.313 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3926357550 2020-07-17T12:03:49.313 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:49.314 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:03:49.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:49.314 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zonegroup": "test-zonegroup", 2020-07-17T12:03:49.315 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:03:49.315 INFO:teuthology.orchestra.run.smithi052.stdout: "period_config": { 2020-07-17T12:03:49.315 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_quota": { 2020-07-17T12:03:49.316 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2020-07-17T12:03:49.316 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2020-07-17T12:03:49.316 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2020-07-17T12:03:49.317 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2020-07-17T12:03:49.317 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2020-07-17T12:03:49.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:49.318 INFO:teuthology.orchestra.run.smithi052.stdout: "user_quota": { 2020-07-17T12:03:49.319 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2020-07-17T12:03:49.319 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2020-07-17T12:03:49.319 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2020-07-17T12:03:49.320 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2020-07-17T12:03:49.320 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2020-07-17T12:03:49.320 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:03:49.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:03:49.321 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:03:49.321 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_name": "test-realm", 2020-07-17T12:03:49.322 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:03:49.322 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:03:49.323 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-07-17T12:03:49.323 INFO:teuthology.orchestra.run:waiting for 300 2020-07-17T12:03:49.367 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2020-07-17T12:03:49.362+0000 7f9493fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 13859) UID: 0 2020-07-17T12:03:49.368 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2020-07-17T12:03:49.362+0000 7f94b76f0100 -1 shutting down 2020-07-17T12:03:53.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-07-17T12:03:55.425 INFO:tasks.rgw.c2.client.0:Stopped 2020-07-17T12:03:55.426 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-07-17T12:03:55.427 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-07-17T12:03:55.430 INFO:tasks.rgw.c2.client.0:Started 2020-07-17T12:03:55.431 INFO:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000 2020-07-17T12:03:55.447 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:03:55.447 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:03:55.451 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi052.front.sepia.ceph.com port 8000: Connection refused 2020-07-17T12:03:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-17T12:03:56.453 INFO:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000 2020-07-17T12:03:56.467 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:03:56.468 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:03:56.472 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-17T12:03:56.473 INFO:teuthology.orchestra.run.smithi052.stdout:anonymous 2020-07-17T12:03:56.474 INFO:tasks.rgw_multisite:creating zone.. 2020-07-17T12:03:56.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--endpoints', 'http://smithi146.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:56.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--endpoints', 'http://smithi146.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:56.475 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 1R9V3H5FL4JONCEX --secret LcR3LFWGav9hRFziuKCRCpXMbs1dgJln --endpoints http://smithi146.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:56.521 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:56.522 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:56.594 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:56.586+0000 7fd67fd100c0 0 failed reading obj info from .rgw.root:zone_info.f2444b35-3e93-4fbd-99ef-52c98595b68f: (2) No such file or directory 2020-07-17T12:03:56.595 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:03:56.586+0000 7fd67fd100c0 0 WARNING: could not read zone params for zone id=f2444b35-3e93-4fbd-99ef-52c98595b68f name=test-zone2 2020-07-17T12:03:56.605 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:56.605 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:03:56.605 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone3", 2020-07-17T12:03:56.606 INFO:teuthology.orchestra.run.smithi146.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-07-17T12:03:56.606 INFO:teuthology.orchestra.run.smithi146.stdout: "control_pool": "test-zone3.rgw.control", 2020-07-17T12:03:56.606 INFO:teuthology.orchestra.run.smithi146.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-07-17T12:03:56.606 INFO:teuthology.orchestra.run.smithi146.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-07-17T12:03:56.606 INFO:teuthology.orchestra.run.smithi146.stdout: "log_pool": "test-zone3.rgw.log", 2020-07-17T12:03:56.607 INFO:teuthology.orchestra.run.smithi146.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-07-17T12:03:56.607 INFO:teuthology.orchestra.run.smithi146.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-07-17T12:03:56.607 INFO:teuthology.orchestra.run.smithi146.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-07-17T12:03:56.607 INFO:teuthology.orchestra.run.smithi146.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-07-17T12:03:56.607 INFO:teuthology.orchestra.run.smithi146.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-07-17T12:03:56.608 INFO:teuthology.orchestra.run.smithi146.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-07-17T12:03:56.608 INFO:teuthology.orchestra.run.smithi146.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-07-17T12:03:56.608 INFO:teuthology.orchestra.run.smithi146.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-07-17T12:03:56.608 INFO:teuthology.orchestra.run.smithi146.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-07-17T12:03:56.608 INFO:teuthology.orchestra.run.smithi146.stdout: "system_key": { 2020-07-17T12:03:56.609 INFO:teuthology.orchestra.run.smithi146.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:03:56.609 INFO:teuthology.orchestra.run.smithi146.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:03:56.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:56.609 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_pools": [ 2020-07-17T12:03:56.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:56.609 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "default-placement", 2020-07-17T12:03:56.610 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:03:56.610 INFO:teuthology.orchestra.run.smithi146.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-07-17T12:03:56.610 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": { 2020-07-17T12:03:56.610 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD": { 2020-07-17T12:03:56.610 INFO:teuthology.orchestra.run.smithi146.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2020-07-17T12:03:56.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:56.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:56.611 INFO:teuthology.orchestra.run.smithi146.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-07-17T12:03:56.611 INFO:teuthology.orchestra.run.smithi146.stdout: "index_type": 0 2020-07-17T12:03:56.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:56.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:56.612 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:56.612 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca" 2020-07-17T12:03:56.612 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:56.613 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-07-17T12:03:57.101 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-07-17T12:03:57.130 INFO:teuthology.orchestra.run.smithi146:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-07-17T12:03:58.120 INFO:teuthology.orchestra.run.smithi146.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-07-17T12:03:58.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:58.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:58.140 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:58.188 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:58.212 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:03:58.213 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:03:58.213 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone3", 2020-07-17T12:03:58.213 INFO:teuthology.orchestra.run.smithi146.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-07-17T12:03:58.213 INFO:teuthology.orchestra.run.smithi146.stdout: "control_pool": "test-zone3.rgw.control", 2020-07-17T12:03:58.214 INFO:teuthology.orchestra.run.smithi146.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-07-17T12:03:58.214 INFO:teuthology.orchestra.run.smithi146.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-07-17T12:03:58.214 INFO:teuthology.orchestra.run.smithi146.stdout: "log_pool": "test-zone3.rgw.log", 2020-07-17T12:03:58.214 INFO:teuthology.orchestra.run.smithi146.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-07-17T12:03:58.215 INFO:teuthology.orchestra.run.smithi146.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-07-17T12:03:58.215 INFO:teuthology.orchestra.run.smithi146.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-07-17T12:03:58.215 INFO:teuthology.orchestra.run.smithi146.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-07-17T12:03:58.215 INFO:teuthology.orchestra.run.smithi146.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-07-17T12:03:58.216 INFO:teuthology.orchestra.run.smithi146.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-07-17T12:03:58.216 INFO:teuthology.orchestra.run.smithi146.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-07-17T12:03:58.216 INFO:teuthology.orchestra.run.smithi146.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-07-17T12:03:58.216 INFO:teuthology.orchestra.run.smithi146.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-07-17T12:03:58.216 INFO:teuthology.orchestra.run.smithi146.stdout: "system_key": { 2020-07-17T12:03:58.217 INFO:teuthology.orchestra.run.smithi146.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:03:58.217 INFO:teuthology.orchestra.run.smithi146.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:03:58.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:58.217 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_pools": [ 2020-07-17T12:03:58.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:03:58.218 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "default-placement", 2020-07-17T12:03:58.218 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:03:58.218 INFO:teuthology.orchestra.run.smithi146.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-07-17T12:03:58.219 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": { 2020-07-17T12:03:58.219 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD": { 2020-07-17T12:03:58.219 INFO:teuthology.orchestra.run.smithi146.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2020-07-17T12:03:58.220 INFO:teuthology.orchestra.run.smithi146.stdout: "compression_type": "random" 2020-07-17T12:03:58.220 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:58.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:03:58.220 INFO:teuthology.orchestra.run.smithi146.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-07-17T12:03:58.220 INFO:teuthology.orchestra.run.smithi146.stdout: "index_type": 0 2020-07-17T12:03:58.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:58.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:03:58.221 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:03:58.221 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "" 2020-07-17T12:03:58.221 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:03:58.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:58.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:03:58.224 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:03:58.258 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:03:58.258 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:03:59.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-07-17T12:04:06.287 INFO:teuthology.orchestra.run.smithi146.stderr:Sending period to new master zone bc359d1d-af48-401e-ac43-335f84c3f1d9 2020-07-17T12:04:06.356 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:04:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 3, 2020-07-17T12:04:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "predecessor_uuid": "68df446e-5a13-4da6-9e83-5e687570550c", 2020-07-17T12:04:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": [], 2020-07-17T12:04:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "period_map": { 2020-07-17T12:04:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "zonegroups": [ 2020-07-17T12:04:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "test-zonegroup", 2020-07-17T12:04:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zonegroup", 2020-07-17T12:04:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "api_name": "", 2020-07-17T12:04:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "is_master": "true", 2020-07-17T12:04:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:04:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:04:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:04:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "hostnames": [], 2020-07-17T12:04:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "hostnames_s3website": [], 2020-07-17T12:04:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:04:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "zones": [ 2020-07-17T12:04:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:04:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone3", 2020-07-17T12:04:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:04:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi146.front.sepia.ceph.com:8001" 2020-07-17T12:04:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:04:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "log_meta": "false", 2020-07-17T12:04:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "log_data": "true", 2020-07-17T12:04:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:04:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "read_only": "false", 2020-07-17T12:04:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_type": "", 2020-07-17T12:04:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from_all": "true", 2020-07-17T12:04:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from": [], 2020-07-17T12:04:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_zone": "" 2020-07-17T12:04:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:04:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone1", 2020-07-17T12:04:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:04:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:04:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:04:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "log_meta": "false", 2020-07-17T12:04:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "log_data": "true", 2020-07-17T12:04:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:04:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "read_only": "false", 2020-07-17T12:04:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_type": "", 2020-07-17T12:04:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from_all": "true", 2020-07-17T12:04:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from": [], 2020-07-17T12:04:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_zone": "" 2020-07-17T12:04:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:04:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone2", 2020-07-17T12:04:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:04:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2020-07-17T12:04:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:04:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "log_meta": "false", 2020-07-17T12:04:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "log_data": "true", 2020-07-17T12:04:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:04:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "read_only": "false", 2020-07-17T12:04:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_type": "", 2020-07-17T12:04:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from_all": "true", 2020-07-17T12:04:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from": [], 2020-07-17T12:04:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_zone": "" 2020-07-17T12:04:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:04:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:04:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_targets": [ 2020-07-17T12:04:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default-placement", 2020-07-17T12:04:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2020-07-17T12:04:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": [ 2020-07-17T12:04:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD" 2020-07-17T12:04:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:04:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:04:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:04:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "default_placement": "default-placement", 2020-07-17T12:04:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:04:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_policy": { 2020-07-17T12:04:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "groups": [] 2020-07-17T12:04:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:04:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:04:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:04:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "short_zone_ids": [ 2020-07-17T12:04:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:04:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 2627409927 2020-07-17T12:04:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:04:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 340514820 2020-07-17T12:04:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:04:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 3926357550 2020-07-17T12:04:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:04:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:04:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zonegroup": "test-zonegroup", 2020-07-17T12:04:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:04:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "period_config": { 2020-07-17T12:04:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_quota": { 2020-07-17T12:04:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:04:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:04:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:04:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:04:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:04:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "user_quota": { 2020-07-17T12:04:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:04:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:04:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:04:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:04:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:04:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:04:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:04:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_name": "test-realm", 2020-07-17T12:04:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:04:06.389 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:04:06.390 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-07-17T12:04:06.390 INFO:teuthology.orchestra.run:waiting for 300 2020-07-17T12:04:06.428 INFO:tasks.rgw.c1.client.1.smithi146.stdout:2020-07-17T12:04:06.422+0000 7f0f22ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 13519) UID: 0 2020-07-17T12:04:06.429 INFO:tasks.rgw.c1.client.1.smithi146.stdout:2020-07-17T12:04:06.422+0000 7f0f3e78a100 -1 shutting down 2020-07-17T12:04:11.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-07-17T12:04:12.496 INFO:tasks.rgw.c1.client.1:Stopped 2020-07-17T12:04:12.496 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-07-17T12:04:12.497 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-07-17T12:04:12.502 INFO:tasks.rgw.c1.client.1:Started 2020-07-17T12:04:12.502 INFO:teuthology.orchestra.run.smithi146:> curl http://smithi146.front.sepia.ceph.com:8001 2020-07-17T12:04:12.519 INFO:teuthology.orchestra.run.smithi146.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:04:12.520 INFO:teuthology.orchestra.run.smithi146.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:04:12.523 INFO:teuthology.orchestra.run.smithi146.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi146.front.sepia.ceph.com port 8001: Connection refused 2020-07-17T12:04:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-17T12:04:13.526 INFO:teuthology.orchestra.run.smithi146:> curl http://smithi146.front.sepia.ceph.com:8001 2020-07-17T12:04:13.541 INFO:teuthology.orchestra.run.smithi146.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:04:13.542 INFO:teuthology.orchestra.run.smithi146.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:04:13.546 INFO:teuthology.orchestra.run.smithi146.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-17T12:04:13.547 INFO:teuthology.orchestra.run.smithi146.stdout:anonymous 2020-07-17T12:04:13.548 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-07-17T12:04:13.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi146.front.sepia.ceph.com:8000', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--url', 'http://smithi146.front.sepia.ceph.com:8000', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:13.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi146.front.sepia.ceph.com:8000', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--url', 'http://smithi146.front.sepia.ceph.com:8000', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:13.549 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi146.front.sepia.ceph.com:8000 --access-key 1R9V3H5FL4JONCEX --secret LcR3LFWGav9hRFziuKCRCpXMbs1dgJln --url http://smithi146.front.sepia.ceph.com:8000 --access-key 1R9V3H5FL4JONCEX --secret LcR3LFWGav9hRFziuKCRCpXMbs1dgJln --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:04:13.598 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:13.599 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:13.645 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2020-07-17T12:04:13.638+0000 7f64c0fd1700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca.control err=-107 2020-07-17T12:04:13.671 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:13.672 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:04:13.672 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-realm", 2020-07-17T12:04:13.672 INFO:teuthology.orchestra.run.smithi052.stdout: "current_period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:13.672 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 2 2020-07-17T12:04:13.672 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:13.673 INFO:tasks.rgw_multisite:creating zone.. 2020-07-17T12:04:13.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:13.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1R9V3H5FL4JONCEX', '--secret', 'LcR3LFWGav9hRFziuKCRCpXMbs1dgJln', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:13.675 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 1R9V3H5FL4JONCEX --secret LcR3LFWGav9hRFziuKCRCpXMbs1dgJln --endpoints http://smithi052.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:04:13.726 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:13.726 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:13.795 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:04:13.790+0000 7f72ffdaf0c0 0 failed reading obj info from .rgw.root:zone_info.0c5f4c70-7e48-49ca-abec-dc0f91059509: (2) No such file or directory 2020-07-17T12:04:13.796 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:04:13.790+0000 7f72ffdaf0c0 0 WARNING: could not read zone params for zone id=0c5f4c70-7e48-49ca-abec-dc0f91059509 name=test-zone3 2020-07-17T12:04:13.796 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:04:13.790+0000 7f72ffdaf0c0 0 failed reading obj info from .rgw.root:zone_info.bc359d1d-af48-401e-ac43-335f84c3f1d9: (2) No such file or directory 2020-07-17T12:04:13.796 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:04:13.790+0000 7f72ffdaf0c0 0 WARNING: could not read zone params for zone id=bc359d1d-af48-401e-ac43-335f84c3f1d9 name=test-zone1 2020-07-17T12:04:13.802 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:04:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone4", 2020-07-17T12:04:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-07-17T12:04:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone4.rgw.control", 2020-07-17T12:04:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-07-17T12:04:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-07-17T12:04:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone4.rgw.log", 2020-07-17T12:04:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-07-17T12:04:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-07-17T12:04:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-07-17T12:04:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-07-17T12:04:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-07-17T12:04:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-07-17T12:04:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-07-17T12:04:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-07-17T12:04:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-07-17T12:04:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2020-07-17T12:04:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:04:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:04:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2020-07-17T12:04:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2020-07-17T12:04:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-07-17T12:04:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2020-07-17T12:04:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2020-07-17T12:04:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2020-07-17T12:04:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-07-17T12:04:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2020-07-17T12:04:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_config": { 2020-07-17T12:04:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "event_retention_days": 7, 2020-07-17T12:04:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "start_with_full_sync": false 2020-07-17T12:04:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca" 2020-07-17T12:04:13.811 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:13.812 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-07-17T12:04:14.996 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-07-17T12:04:15.023 INFO:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-07-17T12:04:16.013 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-07-17T12:04:16.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:16.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:16.042 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:04:16.091 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:16.091 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:16.114 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:04:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone4", 2020-07-17T12:04:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-07-17T12:04:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone4.rgw.control", 2020-07-17T12:04:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-07-17T12:04:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-07-17T12:04:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone4.rgw.log", 2020-07-17T12:04:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-07-17T12:04:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-07-17T12:04:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-07-17T12:04:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-07-17T12:04:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-07-17T12:04:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-07-17T12:04:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-07-17T12:04:16.119 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-07-17T12:04:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-07-17T12:04:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2020-07-17T12:04:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:04:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:04:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2020-07-17T12:04:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2020-07-17T12:04:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-07-17T12:04:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2020-07-17T12:04:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2020-07-17T12:04:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2020-07-17T12:04:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2020-07-17T12:04:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-07-17T12:04:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2020-07-17T12:04:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_config": { 2020-07-17T12:04:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "event_retention_days": 7, 2020-07-17T12:04:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "start_with_full_sync": false 2020-07-17T12:04:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "" 2020-07-17T12:04:16.128 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:16.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:16.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:16.131 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:04:16.169 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:16.170 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:17.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-07-17T12:04:24.192 INFO:teuthology.orchestra.run.smithi052.stderr:Sending period to new master zone bc359d1d-af48-401e-ac43-335f84c3f1d9 2020-07-17T12:04:24.261 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:24.262 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:24.262 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 4, 2020-07-17T12:04:24.262 INFO:teuthology.orchestra.run.smithi052.stdout: "predecessor_uuid": "68df446e-5a13-4da6-9e83-5e687570550c", 2020-07-17T12:04:24.262 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": [], 2020-07-17T12:04:24.263 INFO:teuthology.orchestra.run.smithi052.stdout: "period_map": { 2020-07-17T12:04:24.263 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:24.263 INFO:teuthology.orchestra.run.smithi052.stdout: "zonegroups": [ 2020-07-17T12:04:24.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.263 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "test-zonegroup", 2020-07-17T12:04:24.264 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zonegroup", 2020-07-17T12:04:24.264 INFO:teuthology.orchestra.run.smithi052.stdout: "api_name": "", 2020-07-17T12:04:24.264 INFO:teuthology.orchestra.run.smithi052.stdout: "is_master": "true", 2020-07-17T12:04:24.264 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:04:24.264 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:04:24.264 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:24.265 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames": [], 2020-07-17T12:04:24.265 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames_s3website": [], 2020-07-17T12:04:24.265 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:04:24.265 INFO:teuthology.orchestra.run.smithi052.stdout: "zones": [ 2020-07-17T12:04:24.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.266 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:04:24.266 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone3", 2020-07-17T12:04:24.266 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:04:24.266 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8001" 2020-07-17T12:04:24.267 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:24.267 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:04:24.267 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:04:24.267 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:04:24.267 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:04:24.268 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:04:24.268 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:04:24.268 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:04:24.268 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:04:24.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:24.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.269 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:04:24.269 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone4", 2020-07-17T12:04:24.269 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:04:24.269 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8001" 2020-07-17T12:04:24.269 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:24.270 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:04:24.270 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:04:24.270 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:04:24.271 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:04:24.271 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "pubsub", 2020-07-17T12:04:24.271 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "false", 2020-07-17T12:04:24.271 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [ 2020-07-17T12:04:24.272 INFO:teuthology.orchestra.run.smithi052.stdout: "test-zone1" 2020-07-17T12:04:24.272 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:24.272 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:04:24.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:24.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.273 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:04:24.273 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone1", 2020-07-17T12:04:24.273 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:04:24.274 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:04:24.274 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:24.274 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:04:24.274 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:04:24.275 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:04:24.275 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:04:24.275 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:04:24.275 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:04:24.276 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:04:24.276 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:04:24.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:24.277 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.277 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:04:24.277 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2020-07-17T12:04:24.278 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:04:24.278 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2020-07-17T12:04:24.279 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:24.279 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:04:24.279 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:04:24.280 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:04:24.280 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:04:24.280 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:04:24.281 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:04:24.281 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:04:24.281 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:04:24.281 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:24.282 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:24.282 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_targets": [ 2020-07-17T12:04:24.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.283 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default-placement", 2020-07-17T12:04:24.283 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2020-07-17T12:04:24.283 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": [ 2020-07-17T12:04:24.284 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD" 2020-07-17T12:04:24.284 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:04:24.284 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:24.284 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:24.284 INFO:teuthology.orchestra.run.smithi052.stdout: "default_placement": "default-placement", 2020-07-17T12:04:24.285 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:04:24.285 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_policy": { 2020-07-17T12:04:24.285 INFO:teuthology.orchestra.run.smithi052.stdout: "groups": [] 2020-07-17T12:04:24.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:24.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:24.286 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:04:24.286 INFO:teuthology.orchestra.run.smithi052.stdout: "short_zone_ids": [ 2020-07-17T12:04:24.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.370 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:04:24.370 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 2627409927 2020-07-17T12:04:24.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:24.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.371 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:04:24.371 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3854808555 2020-07-17T12:04:24.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:24.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.371 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:04:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 340514820 2020-07-17T12:04:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:04:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3926357550 2020-07-17T12:04:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:04:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zonegroup": "test-zonegroup", 2020-07-17T12:04:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:04:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: "period_config": { 2020-07-17T12:04:24.374 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_quota": { 2020-07-17T12:04:24.374 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2020-07-17T12:04:24.374 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2020-07-17T12:04:24.374 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2020-07-17T12:04:24.374 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2020-07-17T12:04:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2020-07-17T12:04:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: "user_quota": { 2020-07-17T12:04:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2020-07-17T12:04:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2020-07-17T12:04:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2020-07-17T12:04:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2020-07-17T12:04:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2020-07-17T12:04:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:04:24.377 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_name": "test-realm", 2020-07-17T12:04:24.377 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:24.377 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:24.378 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-07-17T12:04:24.379 INFO:teuthology.orchestra.run:waiting for 300 2020-07-17T12:04:24.423 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2020-07-17T12:04:24.418+0000 7fac26ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 13862) UID: 0 2020-07-17T12:04:24.423 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2020-07-17T12:04:24.418+0000 7fac4a944100 -1 shutting down 2020-07-17T12:04:27.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-07-17T12:04:30.482 INFO:tasks.rgw.c2.client.1:Stopped 2020-07-17T12:04:30.482 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-07-17T12:04:30.483 INFO:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-07-17T12:04:30.486 INFO:tasks.rgw.c2.client.1:Started 2020-07-17T12:04:30.487 INFO:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8001 2020-07-17T12:04:30.505 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:04:30.505 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:04:30.509 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi052.front.sepia.ceph.com port 8001: Connection refused 2020-07-17T12:04:30.510 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-17T12:04:31.511 INFO:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8001 2020-07-17T12:04:31.528 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:04:31.529 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:04:31.533 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 26800 0 --:--:-- --:--:-- --:--:-- 33500 2020-07-17T12:04:31.534 INFO:teuthology.orchestra.run.smithi052.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005f1193cf-115b-test-zone4","HostId":"115b-test-zone4-test-zonegroup"} 2020-07-17T12:04:31.547 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-07-17T12:04:31.548 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-07-17T12:04:31.653 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-07-17T12:04:31.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:31.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:04:31.655 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:04:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:31.900 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:04:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: "user_id": "rgw-multisite-test-user", 2020-07-17T12:04:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: "display_name": "Multisite Test User", 2020-07-17T12:04:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: "email": "", 2020-07-17T12:04:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: "suspended": 0, 2020-07-17T12:04:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "max_buckets": 1000, 2020-07-17T12:04:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "subusers": [], 2020-07-17T12:04:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "keys": [ 2020-07-17T12:04:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "user": "rgw-multisite-test-user", 2020-07-17T12:04:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "access_key": "MR8VKPLODYQ9GBOEK5AJ", 2020-07-17T12:04:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "secret_key": "jCqa665j8cCc2Q08CWmwp7VnBnIC1kglVxxtd7Hr" 2020-07-17T12:04:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:04:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:04:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "swift_keys": [], 2020-07-17T12:04:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": [], 2020-07-17T12:04:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "op_mask": "read, write, delete", 2020-07-17T12:04:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "default_placement": "", 2020-07-17T12:04:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "default_storage_class": "", 2020-07-17T12:04:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_tags": [], 2020-07-17T12:04:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_quota": { 2020-07-17T12:04:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:04:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:04:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:04:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:04:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:04:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "user_quota": { 2020-07-17T12:04:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:04:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:04:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:04:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:04:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:04:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "temp_url_keys": [], 2020-07-17T12:04:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "rgw", 2020-07-17T12:04:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "mfa_ids": [] 2020-07-17T12:04:31.912 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:04:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-07-17T12:04:31.929 INFO:rgw_multi.tests:meta checkpoint 2020-07-17T12:04:31.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:31.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:31.932 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:04:32.012 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:32.012 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:32.412 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:04:32.447 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:04:32.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:32.447 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:04:32.448 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:04:31.890977Z" 2020-07-17T12:04:32.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:32.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:32.448 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:04:32.448 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:04:32.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:32.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:32.449 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:04:32.449 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:04:32.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:04:32.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:04:32.450 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:04:32.450 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:04:32.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:04:32.450 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:04:32.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:04:32.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:32.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:32.452 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:04:32.494 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:32.494 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:32.711 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:32.711 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:04:32.712 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:04:32.712 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:04:32.712 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:04:32.713 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:32.713 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:32.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:32.714 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:04:32.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:32.714 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:04:32.715 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:32.715 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:32.715 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:32.715 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:32.716 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:04:32.716 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:32.716 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:32.717 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:32.717 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:32.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:32.718 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:32.718 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:04:32.718 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:32.719 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:32.719 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:32.719 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:32.719 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:32.720 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:32.720 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:32.720 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:32.721 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:32.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:32.721 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:32.722 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:04:32.722 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:32.722 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:32.723 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:32.723 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:32.723 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:32.724 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:32.724 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:32.724 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:32.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:32.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:32.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:32.725 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:04:32.726 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:32.726 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:32.726 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:32.727 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:32.727 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:32.727 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:32.728 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:32.728 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:32.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:32.729 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:32.729 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:04:32.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:32.730 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:04:32.730 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:04:32.730 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:04:32.731 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:32.731 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:32.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:04:33.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-07-17T12:04:37.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:37.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:37.735 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:04:37.779 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:37.780 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:37.993 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:37.994 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:04:37.994 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:04:37.994 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:04:37.994 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:04:37.995 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:37.995 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:37.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:37.995 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:04:37.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:37.996 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:04:37.996 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:37.996 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:37.996 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:37.996 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:37.997 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:04:37.997 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:37.997 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:37.997 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:37.997 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:37.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:37.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:37.998 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:04:37.998 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:37.998 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:37.998 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:37.999 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:37.999 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:37.999 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:37.999 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:37.999 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:38.000 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:38.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:38.000 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:38.000 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:04:38.000 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:38.000 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:38.001 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:38.001 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:38.001 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:38.001 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:38.001 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:38.002 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:38.002 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:38.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:38.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:38.002 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:04:38.003 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:38.003 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:38.003 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:38.003 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:38.003 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:38.003 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:38.004 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:38.004 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:38.004 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:38.004 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:38.004 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:04:38.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:38.005 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:04:38.005 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:04:38.005 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:04:38.005 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:38.005 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:38.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:04:43.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:43.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:43.008 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:04:43.058 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:43.059 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:43.275 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:43.275 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:04:43.276 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:04:43.276 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:04:43.276 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:04:43.276 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:43.276 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:43.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:43.277 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:04:43.277 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:43.277 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:04:43.278 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:43.278 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:43.278 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:43.278 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:43.279 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:04:43.279 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:43.279 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:43.279 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:43.279 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:43.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:43.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:43.280 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:04:43.281 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:43.281 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:43.281 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:43.281 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:43.282 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:43.282 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:43.282 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:43.282 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:43.282 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:43.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:43.283 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:43.283 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:04:43.283 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:43.284 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:43.284 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:43.284 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:43.284 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:43.285 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:43.285 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:43.285 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:43.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:43.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:43.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:43.286 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:04:43.286 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:43.286 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:43.286 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:43.287 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:43.287 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:43.287 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:43.287 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:43.288 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:43.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:43.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:43.289 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:04:43.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:43.289 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:04:43.289 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:04:43.290 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:04:43.290 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:43.290 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:43.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:04:48.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:48.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:48.294 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:04:48.341 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:48.341 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:48.560 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:48.561 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:04:48.561 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:04:48.561 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:04:48.561 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:04:48.562 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:48.562 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:48.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:48.562 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:04:48.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:48.563 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:04:48.563 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:48.563 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:48.563 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:48.563 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:48.564 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:04:48.564 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:48.564 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:48.564 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:48.564 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:48.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:48.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:48.565 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:04:48.565 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:48.565 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:48.565 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:48.566 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:48.566 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:48.566 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:48.566 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:48.566 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:48.567 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:48.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:48.567 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:48.567 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:04:48.568 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:48.568 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:48.568 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:48.568 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:48.568 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:48.569 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:48.569 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:48.569 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:48.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:48.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:48.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:48.570 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:04:48.570 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:48.571 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:48.571 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:48.571 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:48.571 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:48.571 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:48.572 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:48.572 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:48.572 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:48.572 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:48.572 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:04:48.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:48.573 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:04:48.573 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:04:48.573 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:04:48.573 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:48.573 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:48.574 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:04:53.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:53.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:53.577 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:04:53.616 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:53.617 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:53.839 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:53.840 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:04:53.840 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:04:53.840 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:04:53.841 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:04:53.841 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:53.841 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:53.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:53.842 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:04:53.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:53.842 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:04:53.843 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:53.843 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:53.843 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:53.844 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:53.844 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:04:53.844 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:53.845 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:53.845 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:53.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:53.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:53.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:53.846 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:04:53.847 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:53.847 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:53.847 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:53.848 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:53.848 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:53.848 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:53.849 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:53.849 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:53.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:53.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:53.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:53.850 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:04:53.850 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:53.851 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:53.851 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:53.851 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:53.852 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:53.852 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:53.852 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:53.853 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:53.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:53.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:53.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:53.854 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:04:53.854 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:53.855 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:53.855 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:53.855 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:53.856 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:53.856 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:53.856 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:53.857 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:53.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:53.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:53.858 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:04:53.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:53.858 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:04:53.858 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:04:53.859 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:04:53.859 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:53.859 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:53.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:04:58.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:58.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:04:58.863 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:04:58.904 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:04:58.904 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:04:59.125 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:04:59.126 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:04:59.126 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:04:59.126 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:04:59.126 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:04:59.127 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:04:59.127 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:59.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:59.127 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:04:59.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:59.127 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:04:59.128 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:59.128 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:59.128 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:59.128 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:59.129 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:04:59.129 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:59.129 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:59.129 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:59.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:59.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:59.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:59.130 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:04:59.130 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:59.131 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:59.131 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:59.131 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:59.131 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:59.131 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:59.132 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:59.132 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:59.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:59.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:59.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:59.133 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:04:59.133 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:59.133 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:59.133 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:59.134 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:59.134 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:59.134 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:59.135 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:59.135 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:59.135 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:59.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:59.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:04:59.136 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:04:59.137 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:04:59.137 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:04:59.137 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:04:59.137 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:04:59.138 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:04:59.138 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:04:59.138 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:04:59.139 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:04:59.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:59.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:59.139 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:04:59.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:04:59.140 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:04:59.140 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:04:59.140 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:04:59.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:04:59.141 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:04:59.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:04.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:04.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:04.146 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:04.187 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:04.188 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:04.408 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:04.422 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:04.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:09.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:09.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:09.426 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:09.469 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:09.470 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:09.699 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:09.699 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:09.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:09.701 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:09.702 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:09.703 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:09.704 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:09.705 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:09.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:09.707 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:09.708 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:09.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:09.710 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:09.711 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:09.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:09.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:09.714 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:09.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:14.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:14.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:14.723 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:14.773 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:14.773 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:14.990 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:14.990 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:14.990 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:14.990 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:14.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:14.992 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:14.992 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:14.992 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:14.992 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:14.995 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:14.995 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:14.995 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:14.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:14.995 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:14.996 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:14.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:15.006 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:15.008 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:20.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:20.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:20.011 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:20.056 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:20.057 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:20.261 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:20.276 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:20.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:25.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:25.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:25.279 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:25.319 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:25.548 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:25.548 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:25.549 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:25.549 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:25.549 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:25.550 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:25.550 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:25.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:25.551 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:25.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:25.552 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:25.552 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:25.552 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:25.553 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:25.553 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:25.553 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:25.555 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:25.555 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:25.555 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:25.556 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:25.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:25.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:25.558 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:25.558 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:25.558 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:25.559 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:25.559 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:25.559 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:25.560 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:25.560 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:25.560 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:25.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:25.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:25.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:25.562 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:25.565 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:25.565 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:25.568 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:25.568 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:25.569 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:25.569 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:25.569 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:25.569 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:25.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:25.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:25.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:25.571 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:25.571 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:25.571 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:25.572 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:25.572 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:25.572 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:25.573 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:25.574 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:25.574 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:25.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:25.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:25.574 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:25.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:25.575 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:25.575 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:25.575 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:25.575 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:25.576 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:25.576 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:30.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:30.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:30.579 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:30.619 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:30.620 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:30.841 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:30.841 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:30.842 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:30.842 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:30.842 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:30.842 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:30.843 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:30.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:30.843 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:30.843 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:30.843 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:30.844 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:30.844 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:30.844 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:30.844 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:30.845 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:30.845 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:30.845 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:30.845 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:30.846 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:30.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:30.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:30.846 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:30.846 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:30.847 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:30.847 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:30.847 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:30.847 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:30.848 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:30.848 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:30.848 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:30.848 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:30.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:30.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:30.849 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:30.849 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:30.849 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:30.850 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:30.850 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:30.850 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:30.850 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:30.851 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:30.851 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:30.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:30.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:30.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:30.852 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:30.852 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:30.853 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:30.853 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:30.854 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:30.854 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:30.855 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:30.855 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:30.856 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:30.856 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:30.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:30.857 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:30.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:30.858 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:30.858 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:30.858 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:30.859 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:30.859 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:30.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:35.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:35.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:35.864 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:35.908 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:35.909 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:36.096 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:36.096 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:36.097 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:36.097 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:36.097 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:36.098 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:36.098 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:36.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:36.099 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:36.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:36.099 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:36.099 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:36.100 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:36.100 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:36.100 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:36.101 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:36.101 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:36.101 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:36.102 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:36.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:36.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:36.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:36.103 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:36.103 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:36.104 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:36.104 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:36.104 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:36.105 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:36.105 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:36.105 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:36.106 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:36.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:36.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:36.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:36.107 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:36.107 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:36.107 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:36.108 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:36.108 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:36.108 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:36.109 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:36.109 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:36.109 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:36.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:36.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:36.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:36.111 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:36.111 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:36.112 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:36.112 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:36.112 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:36.113 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:36.113 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:36.113 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:36.113 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:36.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:36.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:36.114 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:36.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:36.114 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:36.115 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:36.115 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:36.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:36.115 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:36.116 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:41.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:41.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:41.119 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:41.160 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:41.160 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:41.376 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:41.377 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:41.377 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:41.377 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:41.378 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:41.378 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:41.378 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:41.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:41.378 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:41.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:41.379 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:41.379 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:41.379 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:41.380 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:41.380 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:41.380 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:41.380 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:41.380 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:41.381 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:41.381 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:41.381 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:41.382 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:41.382 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:41.382 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:41.382 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:41.382 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:41.383 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:41.383 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:41.383 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:41.383 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:41.383 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:41.384 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:41.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:41.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:41.384 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:41.385 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:41.385 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:41.385 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:41.385 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:41.385 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:41.386 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:41.386 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:41.386 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:41.386 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:41.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:41.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:41.387 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:41.387 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:41.387 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:41.387 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:41.388 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:41.388 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:41.388 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:41.388 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:41.388 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:41.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:41.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:41.389 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:41.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:41.389 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:41.390 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:41.390 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:41.390 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:41.390 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:41.392 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:46.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:46.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:46.394 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:46.435 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:46.435 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:46.655 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:46.655 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:46.656 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:46.656 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:46.656 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:46.656 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:46.656 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:46.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:46.657 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:46.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:46.657 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:46.658 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:46.658 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:46.658 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:46.658 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:46.659 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:46.659 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:46.659 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:46.659 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:46.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:46.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:46.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:46.660 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:46.660 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:46.660 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:46.661 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:46.661 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:46.661 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:46.661 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:46.661 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:46.662 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:46.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:46.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:46.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:46.663 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:46.663 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:46.664 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:46.664 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:46.664 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:46.664 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:46.665 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:46.665 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:46.665 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:46.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:46.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:46.666 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:46.666 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:46.666 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:46.667 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:46.667 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:46.667 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:46.667 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:46.668 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:46.668 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:46.668 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:46.669 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:46.669 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:46.669 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:46.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:46.670 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:46.670 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:46.670 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:46.671 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:46.671 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:46.672 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:51.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:51.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:51.675 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:51.717 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:51.718 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:51.930 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:51.943 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:51.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:05:56.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:56.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:05:56.947 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:05:56.989 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:05:56.990 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:05:57.220 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:05:57.220 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:05:57.221 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:05:57.221 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:05:57.221 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:05:57.222 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:05:57.222 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:57.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:57.223 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:05:57.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:57.223 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:05:57.224 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:57.224 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:57.224 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:57.225 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:57.225 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:05:57.225 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:57.226 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:57.226 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:57.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:57.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:57.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:57.232 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:05:57.232 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:57.232 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:57.233 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:57.233 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:57.233 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:57.234 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:57.234 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:57.234 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:57.235 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:57.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:57.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:57.236 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:05:57.236 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:57.236 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:57.237 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:57.237 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:57.237 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:57.238 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:57.238 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:57.238 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:57.239 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:57.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:57.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:05:57.240 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:05:57.240 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:05:57.240 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:05:57.241 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:05:57.258 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:05:57.259 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:05:57.259 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:05:57.260 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:05:57.260 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:05:57.261 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:57.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:57.262 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:05:57.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:05:57.263 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:05:57.263 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:05:57.264 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:05:57.264 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:05:57.264 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:05:57.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:06:02.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:02.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:02.268 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:02.310 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:02.311 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:02.527 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:02.528 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:02.528 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:02.528 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:02.528 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:02.528 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:02.529 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:02.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:02.529 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:02.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:02.529 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:02.531 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:02.531 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:02.531 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:02.531 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:02.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:02.533 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:02.533 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:02.533 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:02.533 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:02.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:02.536 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:02.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:02.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:02.536 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:02.538 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:02.538 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:02.538 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:02.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:02.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:02.539 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:02.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:02.539 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:02.539 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:02.540 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:02.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:02.540 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:02.542 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:06:07.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:07.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:07.544 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:07.591 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:07.592 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:07.809 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:07.810 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:07.810 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:07.810 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:07.810 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:07.811 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:07.811 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:07.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:07.811 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:07.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:07.812 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:07.812 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:07.812 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:07.812 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:07.812 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:07.813 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:07.813 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:07.813 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:07.813 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:07.814 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:07.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:07.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:07.814 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:07.814 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:07.815 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:07.815 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:07.815 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:07.815 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:07.815 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:07.816 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:07.816 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:07.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:07.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:07.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:07.817 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:07.817 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:07.817 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:07.817 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:07.817 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:07.818 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:07.818 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:07.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:07.818 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:07.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:07.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:07.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:07.821 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:07.821 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:07.821 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:07.822 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:07.822 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:07.822 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:07.823 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:07.823 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:07.823 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:07.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:07.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:07.824 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:07.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:07.825 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:07.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:06:12.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:12.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:12.832 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:12.876 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:12.877 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:13.093 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:13.093 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:13.094 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:13.094 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:13.094 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:13.094 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:13.094 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:13.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:13.095 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:13.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:13.095 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:13.095 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:13.096 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:13.096 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:13.096 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:13.096 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:13.096 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:13.097 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:13.098 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:13.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:13.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:13.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:13.099 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:13.099 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:13.099 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:13.099 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:13.100 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:13.100 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:13.100 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:13.100 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:13.101 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:13.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:13.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:13.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:13.101 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:13.102 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:13.102 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:13.102 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:13.102 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:13.102 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:13.103 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:13.103 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:13.103 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:13.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:13.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:13.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:13.104 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:13.104 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:13.104 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:13.105 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:13.105 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:13.105 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:13.105 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:13.105 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:13.105 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:13.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:13.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:13.106 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:13.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:13.114 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:13.114 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:13.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:13.114 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:13.115 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:06:18.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:18.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:18.119 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:18.159 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:18.160 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:18.389 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:18.389 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:18.390 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:18.390 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:18.391 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:18.391 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:18.391 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:18.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:18.392 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:18.392 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:18.393 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:18.393 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:18.393 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:18.394 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:18.394 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:18.394 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:18.395 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:18.395 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:18.396 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:18.396 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:18.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:18.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:18.397 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:18.397 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:18.398 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:18.398 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:18.398 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:18.399 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:18.399 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:18.400 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:18.400 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:18.400 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:18.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:18.401 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:18.401 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:18.410 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:18.410 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:18.410 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:18.411 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:18.411 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:18.411 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:18.411 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:18.411 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:18.412 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:18.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:18.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:18.412 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:18.412 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:18.413 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:18.413 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:18.413 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:18.413 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:18.414 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:18.414 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:18.414 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:18.414 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:18.414 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:18.414 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:18.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:18.415 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:18.415 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:18.415 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:18.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:18.416 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:18.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:06:23.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:23.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:23.418 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:23.458 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:23.459 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:23.683 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:23.688 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:23.688 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:23.688 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:23.688 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:23.688 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:23.689 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:23.689 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:23.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:23.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:23.689 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:23.690 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:23.690 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:23.690 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:23.690 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:23.690 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:23.690 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:23.691 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:23.691 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:23.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:23.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:23.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:23.692 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:23.692 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:23.692 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:23.692 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:23.692 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:23.692 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:23.693 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:23.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:23.693 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:23.693 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:23.693 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:23.695 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:23.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:23.695 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:23.695 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:23.695 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:23.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:23.696 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:23.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987471.890977_45.1 target= 2020-07-17T12:06:28.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:28.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:28.700 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:28.742 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:28.742 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:28.967 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:28.968 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:28.968 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:28.968 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:28.969 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:28.969 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:28.969 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:28.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:28.969 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:28.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:28.970 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:28.970 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:28.970 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:28.971 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:28.971 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:28.971 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:28.972 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:28.972 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:28.972 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:28.973 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:28.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:28.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:28.974 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:28.975 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:28.975 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:28.975 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:28.975 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:28.976 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:28.976 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:28.976 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:28.977 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:28.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:28.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:28.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:28.978 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:28.978 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:28.978 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:28.978 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:28.979 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:28.979 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:28.979 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:28.980 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:28.980 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:28.980 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:28.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:28.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:28.981 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:28.981 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:28.981 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:28.982 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:28.982 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:28.982 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:28.982 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:28.982 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:28.982 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:28.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:28.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:28.983 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:28.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:28.983 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:28.984 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:28.984 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:28.984 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:28.984 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:28.986 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:06:28.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:06:28.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:28.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:28.988 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:29.032 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:29.032 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:29.264 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:29.265 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:06:29.265 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:06:29.265 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:06:29.265 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:06:29.265 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:29.266 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:29.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:29.266 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:29.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:29.266 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:29.267 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:29.267 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:29.267 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:29.267 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:29.267 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:06:29.268 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:29.268 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:29.268 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:29.268 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:29.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:29.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:29.269 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:29.269 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:29.269 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:29.269 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:29.270 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:29.270 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:29.270 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:29.270 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:29.270 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:29.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:29.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:29.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:29.271 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:29.271 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:29.272 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:29.272 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:29.272 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:29.272 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:29.272 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:29.273 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:29.273 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:29.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:29.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:29.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:29.274 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:29.275 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:29.275 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:29.275 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:29.275 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:29.275 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:29.276 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:29.276 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:29.276 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:29.276 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:29.276 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:29.277 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:29.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:29.277 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:06:29.277 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:06:29.277 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:06:29.278 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:29.278 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:29.278 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:06:29.279 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:06:29.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:29.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:29.280 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:29.320 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:29.321 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:29.589 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:29.589 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:29.589 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:29.590 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:29.590 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:29.590 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:29.590 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:29.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:29.591 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:29.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:29.591 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:29.591 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:29.591 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:29.592 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:29.592 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:29.592 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:29.592 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:29.592 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:29.593 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:29.593 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:29.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:29.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:29.593 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:29.594 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:29.594 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:29.594 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:29.594 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:29.594 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:29.594 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:29.595 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:29.595 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:29.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:29.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:29.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:29.596 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:29.596 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:29.596 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:29.596 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:29.597 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:29.597 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:29.597 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:29.597 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:29.597 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:29.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:29.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:29.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:29.600 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:29.600 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:29.600 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:29.600 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:29.601 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:29.601 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:29.601 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:29.602 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:29.602 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:29.602 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:29.602 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:29.602 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:29.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:29.603 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:29.603 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:29.603 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:29.604 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:29.604 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:29.605 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:06:29.605 INFO:rgw_multi.tests:meta checkpoint 2020-07-17T12:06:29.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:29.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:29.607 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:29.650 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:29.651 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:29.844 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:06:29.875 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:04:31.890977Z" 2020-07-17T12:06:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:29.880 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:29.882 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:06:29.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:29.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:29.883 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:29.923 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:29.923 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:30.138 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:30.138 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:30.138 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:30.139 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:30.139 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:30.139 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:30.139 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.140 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:30.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:30.140 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:30.140 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:30.140 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:30.140 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:30.141 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:30.141 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:30.141 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:30.141 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:30.141 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:30.142 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:30.142 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:30.143 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:30.143 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:30.143 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:30.143 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:30.143 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:30.143 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:30.144 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:30.144 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:30.144 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:30.145 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:30.145 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:30.145 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:30.145 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:30.146 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:30.146 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:30.146 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:30.147 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:30.147 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:30.147 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:30.147 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:30.148 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:30.148 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:30.148 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:30.148 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:30.148 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.170 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.174 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.176 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:30.177 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:06:30.178 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:06:30.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:30.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:30.179 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:30.220 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:30.220 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:30.440 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:30.441 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:06:30.441 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:06:30.441 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:06:30.441 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:06:30.442 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:30.442 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:30.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:30.442 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:30.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:30.443 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:30.443 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:30.443 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:30.443 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:30.444 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:30.444 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:06:30.444 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:30.444 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:30.445 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:30.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:30.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:30.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:30.446 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:30.446 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:30.446 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:30.447 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:30.447 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:30.447 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:30.447 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:30.448 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:30.448 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:30.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:30.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:30.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:30.449 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:30.449 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:30.449 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:30.449 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:30.449 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:30.450 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:30.450 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:30.450 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:30.450 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:30.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:30.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:30.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:30.457 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:30.457 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:30.457 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:30.458 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:30.458 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:30.458 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:30.459 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:30.459 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:30.459 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:30.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:30.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:30.461 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:06:30.467 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:06:30.467 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:06:30.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:30.468 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:30.469 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:06:30.469 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:06:30.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:30.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:30.471 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:30.512 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:30.512 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:30.782 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:30.782 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:30.782 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:30.783 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:30.783 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:30.783 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:30.783 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.784 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:30.784 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:30.784 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:30.784 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:30.785 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:30.785 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:30.785 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:30.785 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:30.785 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:30.786 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:30.786 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:30.787 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:30.787 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:30.794 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:30.794 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:30.794 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:30.795 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:30.795 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:30.795 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:30.796 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.796 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:30.797 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:30.797 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:30.797 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:30.798 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:30.798 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:30.798 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:30.798 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:30.799 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:30.799 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.800 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:30.801 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:30.801 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:30.801 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:30.801 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:30.802 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:30.802 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:30.802 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:30.803 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:30.803 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:30.803 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.803 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.803 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:30.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:30.804 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:30.804 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:30.804 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:30.804 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:30.805 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:30.806 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:06:30.806 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-07-17T12:06:30.816 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-1 2020-07-17T12:06:34.332 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-2 2020-07-17T12:06:38.131 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-3 2020-07-17T12:06:38.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:38.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:38.204 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:38.242 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:38.243 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:38.441 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:06:38.465 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:04:31.890977Z" 2020-07-17T12:06:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987598.180439_53.1", 2020-07-17T12:06:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:38.180439Z" 2020-07-17T12:06:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:38.470 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:38.472 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:06:38.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:38.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:38.474 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:38.517 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:38.518 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:38.740 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:38.740 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:38.741 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:38.741 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:38.741 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:38.741 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:38.741 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:38.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:38.742 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:38.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:38.742 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:38.743 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:38.743 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:38.743 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:38.743 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:38.744 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:38.744 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:38.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:38.744 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:38.745 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:38.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:38.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:38.746 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:38.746 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:38.746 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:38.746 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987598.180439_53.1", 2020-07-17T12:06:38.746 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:38.747 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:38.747 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:38.747 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.180439Z", 2020-07-17T12:06:38.747 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:38.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:38.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:38.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:38.748 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:38.748 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:38.748 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:38.749 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:38.749 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:38.749 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:38.749 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:38.758 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:38.758 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:38.758 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:38.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:38.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:38.759 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:38.759 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:38.759 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:38.759 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:38.759 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:38.760 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:38.760 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:38.760 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:38.760 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:38.760 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:38.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:38.761 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:38.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:38.761 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:38.761 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:38.762 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:38.762 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:38.762 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:38.763 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:06:38.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:06:38.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:38.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:38.765 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:38.808 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:38.808 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:39.024 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:39.024 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:06:39.024 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:06:39.024 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:06:39.025 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:06:39.025 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:39.025 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:39.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:39.025 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:39.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:39.026 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:39.026 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:39.026 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:39.026 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:39.027 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:39.027 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:06:39.027 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:39.027 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:39.027 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:39.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:39.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:39.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:39.028 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:39.028 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:39.028 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:39.029 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987598.180439_53.1", 2020-07-17T12:06:39.029 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:39.029 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:39.029 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:39.029 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:38.180439Z", 2020-07-17T12:06:39.030 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:39.030 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:39.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:39.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:39.030 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:39.030 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:39.031 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:39.031 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:39.031 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:39.031 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:39.031 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:39.032 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:39.032 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:39.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:39.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:39.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:39.032 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:39.033 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:39.033 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:39.033 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:39.033 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:39.033 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:39.034 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:39.034 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:39.034 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:39.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:39.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:39.035 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:39.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:39.035 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:06:39.035 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:06:39.036 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:06:39.036 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:39.036 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:39.037 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:06:39.038 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:06:39.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:39.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:39.040 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:39.080 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:39.081 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:39.354 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:39.354 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:39.355 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:39.355 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:39.355 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:39.355 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:39.355 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987598.180439_53.1", 2020-07-17T12:06:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.180439Z", 2020-07-17T12:06:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:39.363 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:39.363 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:39.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:39.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:39.363 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:39.365 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:39.365 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:39.365 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:39.365 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:39.365 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:39.366 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:39.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:39.367 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:39.367 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:39.367 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:39.367 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:39.367 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:39.368 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:06:39.414 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-07-17T12:06:39.416 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-4 2020-07-17T12:06:39.451 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-5 2020-07-17T12:06:39.500 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-6 2020-07-17T12:06:39.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:39.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:39.559 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:39.596 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:39.596 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:39.806 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:06:39.831 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:39.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:39.831 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:39.832 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:04:31.890977Z" 2020-07-17T12:06:39.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:39.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:39.832 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987599.533990_65.1", 2020-07-17T12:06:39.832 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:39.533990Z" 2020-07-17T12:06:39.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:39.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:39.833 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:39.833 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:39.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:39.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:39.834 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:39.834 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:39.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:39.834 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:39.835 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:06:39.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:39.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:39.837 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:39.879 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:39.880 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:40.079 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:40.079 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:40.079 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:40.079 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:40.080 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:40.080 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:40.080 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.080 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:40.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:40.081 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:40.081 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:40.081 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:40.081 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:40.082 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:40.082 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:40.082 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:40.082 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:40.082 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:40.083 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:40.083 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:40.084 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:40.084 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987599.533990_65.1", 2020-07-17T12:06:40.084 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:40.084 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:40.084 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:40.085 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.533990Z", 2020-07-17T12:06:40.085 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.085 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:40.086 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:40.086 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:40.086 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:40.086 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:40.087 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:40.087 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:40.087 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:40.087 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:40.087 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:40.088 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:40.088 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:40.089 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:40.089 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:40.089 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:40.089 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:40.089 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:40.090 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:40.090 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.090 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:40.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.091 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:40.091 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:40.091 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:40.091 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.092 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:40.096 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:06:40.097 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:06:40.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:40.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:40.099 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:40.140 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:40.141 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:40.337 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:40.337 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:06:40.338 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:06:40.338 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:06:40.338 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:06:40.339 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:40.339 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:40.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:40.339 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:40.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:40.340 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:40.340 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:40.340 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:40.340 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:40.341 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:40.341 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:06:40.341 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:40.341 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:40.341 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:40.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:40.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:40.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:40.342 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:40.343 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:40.343 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:40.343 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987599.533990_65.1", 2020-07-17T12:06:40.344 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:40.344 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:40.344 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:40.345 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.533990Z", 2020-07-17T12:06:40.345 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:40.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:40.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:40.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:40.346 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:40.346 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:40.347 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:40.347 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:40.347 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:40.348 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:40.348 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:40.348 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:40.348 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:40.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:40.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:40.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:40.350 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:40.350 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:40.350 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:40.351 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:40.351 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:40.351 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:40.352 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:40.352 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:40.352 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:40.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:40.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:40.353 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:40.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:40.354 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:06:40.354 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:06:40.355 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:06:40.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:40.355 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:40.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:06:40.358 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:06:40.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:40.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:40.361 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:40.403 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:40.403 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:40.680 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:40.680 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:40.680 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:40.681 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:40.681 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:40.681 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:40.682 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.682 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:40.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:40.683 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:40.691 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:40.692 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:40.692 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:40.692 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:40.693 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:40.693 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:40.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:40.694 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:40.695 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:40.695 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:40.696 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:40.696 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987599.533990_65.1", 2020-07-17T12:06:40.696 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:40.697 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:40.697 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:40.697 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.533990Z", 2020-07-17T12:06:40.698 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.698 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:40.699 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:40.699 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:40.699 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:40.700 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:40.700 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:40.700 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:40.701 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:40.702 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:40.702 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.702 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:40.703 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:40.704 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:40.704 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:40.704 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:40.705 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:40.705 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:40.705 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:40.706 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:40.706 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:40.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.707 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.707 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:40.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:40.708 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:40.708 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:40.708 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:40.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:40.709 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:40.711 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:06:41.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:41.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:41.066 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:41.102 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:41.102 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:41.279 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:06:41.308 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:41.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:41.309 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:41.309 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:04:31.890977Z" 2020-07-17T12:06:41.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:41.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:41.310 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987599.533990_65.1", 2020-07-17T12:06:41.311 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:39.533990Z" 2020-07-17T12:06:41.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:41.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:41.312 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:41.312 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:41.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:41.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:41.312 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:41.313 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:41.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:41.313 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:41.314 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:06:41.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:41.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:41.316 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:41.363 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:41.364 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:41.590 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:41.590 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:41.590 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:41.591 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:41.591 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:41.591 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:41.591 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:41.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:41.592 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:41.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:41.592 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:41.592 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:41.592 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:41.593 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:41.593 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:41.593 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:41.593 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:41.593 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:41.594 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:41.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:41.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:41.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:41.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:41.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:41.595 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:41.596 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987599.533990_65.1", 2020-07-17T12:06:41.596 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:41.596 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:41.597 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:41.597 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.533990Z", 2020-07-17T12:06:41.597 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:41.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:41.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:41.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:41.599 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:41.599 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:41.599 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:41.599 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:41.600 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:41.600 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:41.600 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:41.601 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:41.601 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:41.601 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:41.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:41.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:41.602 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:41.602 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:41.603 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:41.603 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:41.603 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:41.604 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:41.604 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:41.604 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:41.604 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:41.605 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:41.605 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:41.608 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:41.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:41.609 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:41.609 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:41.609 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:41.609 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:41.610 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:41.611 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:06:41.613 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:06:41.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:41.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:41.614 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:41.655 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:41.655 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:41.895 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:41.896 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:06:41.896 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:06:41.896 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:06:41.896 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:06:41.896 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:41.898 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:41.898 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:41.898 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:41.898 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:06:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987599.533990_65.1", 2020-07-17T12:06:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.533990Z", 2020-07-17T12:06:41.902 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:41.902 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:41.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:41.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:41.902 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:41.904 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:41.906 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:41.906 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:41.906 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:41.906 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:41.906 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:41.908 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:41.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:41.908 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:06:41.908 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:06:41.908 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:06:41.909 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:41.909 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:41.910 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:06:41.927 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:06:41.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:41.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:41.929 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:41.964 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:41.965 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:42.232 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:42.233 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:42.233 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:42.233 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:42.233 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:42.233 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:42.234 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.234 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:42.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:42.235 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:42.235 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:42.235 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:42.236 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:42.236 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:42.236 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:42.237 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:42.237 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:42.237 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.237 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:42.238 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:42.238 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:42.238 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:42.239 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987599.533990_65.1", 2020-07-17T12:06:42.239 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:42.240 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:42.240 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:42.240 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.533990Z", 2020-07-17T12:06:42.240 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:42.241 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:42.242 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:42.242 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:42.242 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:42.242 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:42.242 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:42.243 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:42.243 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:42.243 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.243 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:42.244 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:42.244 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:42.244 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:42.244 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:42.244 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:42.245 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:42.245 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:42.245 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:42.245 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.245 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.246 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.246 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:42.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.246 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:42.246 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:42.246 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:42.247 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.247 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:42.248 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:06:42.297 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-07-17T12:06:42.298 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-7 2020-07-17T12:06:42.338 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-8 2020-07-17T12:06:42.401 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-9 2020-07-17T12:06:42.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:42.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:42.473 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:42.507 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:42.508 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:06:42.698 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:04:31.890977Z" 2020-07-17T12:06:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987602.439076_78.1", 2020-07-17T12:06:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:42.439076Z" 2020-07-17T12:06:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:42.702 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:42.703 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:06:42.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:42.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:42.705 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:42.745 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:42.745 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:42.965 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:42.966 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:42.966 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:42.966 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:42.966 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:42.967 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:42.967 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.967 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:42.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:42.968 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:42.968 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:42.968 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:42.968 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:42.968 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:42.969 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:42.969 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:42.969 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:42.969 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.969 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:42.970 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:42.970 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:42.970 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:42.971 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987602.439076_78.1", 2020-07-17T12:06:42.971 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:42.971 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:42.971 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:42.971 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.439076Z", 2020-07-17T12:06:42.972 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.972 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:42.972 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:42.973 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:42.973 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:42.973 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:42.973 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:42.973 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:42.974 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:42.974 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:42.974 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.974 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:42.975 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:42.975 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:42.977 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:42.978 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:42.978 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:42.978 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:42.978 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:42.979 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:42.979 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:42.979 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.979 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.980 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:42.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:42.980 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:42.980 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:42.980 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:42.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:42.981 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:42.982 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:06:42.982 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:06:42.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:42.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:42.984 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:43.024 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:43.024 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:43.251 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:43.251 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:06:43.251 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:06:43.252 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:06:43.252 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:06:43.252 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:43.253 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:43.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:43.253 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:43.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:43.254 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:43.254 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:43.255 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:43.255 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:43.255 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:43.256 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:06:43.256 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:43.256 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:43.257 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:43.257 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:43.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:43.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:43.258 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:43.258 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:43.259 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:43.259 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987602.439076_78.1", 2020-07-17T12:06:43.259 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:43.260 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:43.261 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:43.261 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.439076Z", 2020-07-17T12:06:43.262 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:43.262 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:43.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:43.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:43.263 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:43.263 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:43.263 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:43.264 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:43.264 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:43.264 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:43.265 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:43.265 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:43.265 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:43.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:43.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:43.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:43.267 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:43.267 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:43.268 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:43.268 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:43.268 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:43.269 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:43.269 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:43.270 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:43.270 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:43.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:43.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:43.271 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:43.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:43.273 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:06:43.273 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:06:43.274 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:06:43.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:43.275 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:43.276 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:06:43.277 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:06:43.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:43.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:43.282 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:43.322 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:43.323 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:43.587 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:43.588 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:43.588 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:43.589 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:43.589 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:43.589 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:43.590 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:43.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:43.590 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:43.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:43.591 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:43.591 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:43.591 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:43.592 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:43.592 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:43.592 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:43.593 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:43.593 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:43.593 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:43.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:43.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:43.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:43.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:43.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:43.595 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:43.596 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987602.439076_78.1", 2020-07-17T12:06:43.596 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:43.596 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:43.597 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:43.597 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.439076Z", 2020-07-17T12:06:43.597 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:43.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:43.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:43.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:43.599 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:43.599 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:43.599 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:43.599 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:43.600 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:43.600 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:43.600 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:43.601 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:43.601 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:43.601 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:43.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:43.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:43.606 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:43.607 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:43.607 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:43.607 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:43.608 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:43.608 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:43.608 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:43.609 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:43.609 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:43.609 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:43.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:43.610 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:43.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:43.611 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:43.611 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:43.611 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:43.612 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:43.612 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:43.618 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:06:43.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:43.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:43.961 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:43.999 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:43.999 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:44.180 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:06:44.209 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:44.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:44.210 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:44.210 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:04:31.890977Z" 2020-07-17T12:06:44.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:44.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:44.211 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:44.211 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:06:44.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:44.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:44.212 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:44.212 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:44.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:44.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:44.213 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:44.214 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:44.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:44.214 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:44.220 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:06:44.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:44.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:44.223 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:44.269 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:44.269 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:44.490 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:44.490 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:44.491 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:44.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:44.493 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:44.494 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:44.495 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:44.497 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:44.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:44.499 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:06:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:44.517 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:44.519 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:06:44.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:06:44.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:44.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:44.522 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:44.563 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:44.564 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:44.785 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:44.786 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:06:44.786 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:06:44.786 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:06:44.786 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:06:44.787 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:44.787 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:44.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:44.787 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:44.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:44.788 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:44.788 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:44.788 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:44.788 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:44.789 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:44.789 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:06:44.789 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:44.789 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:44.790 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:44.790 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:44.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:44.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:44.791 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:44.791 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:44.791 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:44.792 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:44.792 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:44.792 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:44.792 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:44.792 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:06:44.793 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:44.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:44.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:44.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:44.793 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:44.794 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:44.794 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:44.794 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:44.794 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:44.794 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:44.795 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:44.795 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:44.797 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:44.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:44.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:44.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:44.802 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:44.803 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:44.803 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:44.803 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:44.804 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:44.804 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:44.804 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:44.804 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:44.805 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:44.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:44.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:44.806 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:44.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:44.806 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:06:44.806 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:06:44.807 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:06:44.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:44.807 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:44.809 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:06:44.810 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:06:44.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:44.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:44.812 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:44.852 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:44.852 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:45.117 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:45.118 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:45.118 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:45.118 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:45.118 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:45.119 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:45.119 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:45.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:45.120 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:45.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:45.120 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:45.121 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:45.121 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:45.121 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987471.890977_45.1", 2020-07-17T12:06:45.121 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:45.122 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:45.122 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:45.122 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977Z", 2020-07-17T12:06:45.123 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:45.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:45.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:45.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:45.124 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:45.125 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:45.125 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:45.125 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:45.125 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:45.126 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:45.126 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:45.126 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:06:45.127 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:45.127 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:45.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:45.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:45.128 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:45.128 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:45.128 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:45.128 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:45.129 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:45.129 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:45.129 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:45.129 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:45.130 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:45.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:45.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:45.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:45.131 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:45.131 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:45.131 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:45.132 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:45.132 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:45.132 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:45.132 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:45.133 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:45.133 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:45.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:45.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:45.134 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:45.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:45.134 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:45.135 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:45.135 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:45.135 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:45.135 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:45.136 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:06:45.159 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-07-17T12:06:45.160 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-10 2020-07-17T12:06:45.193 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-11 2020-07-17T12:06:45.247 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-12 2020-07-17T12:06:45.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:45.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:45.912 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:45.953 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:45.954 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:46.195 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:06:46.226 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:46.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:46.227 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987605.278144_59.1", 2020-07-17T12:06:46.227 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:45.278144Z" 2020-07-17T12:06:46.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:46.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:46.228 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:46.228 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:06:46.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:46.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:46.229 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:46.229 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:46.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:46.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:46.230 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:46.230 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:46.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:46.230 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:46.231 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:06:46.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:46.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:46.233 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:46.283 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:46.283 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:46.450 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:46.451 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:46.451 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:46.451 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:46.452 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:46.452 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:46.452 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:46.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:46.452 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:46.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:46.453 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:46.453 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:46.453 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:46.453 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987605.278144_59.1", 2020-07-17T12:06:46.453 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:46.454 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:46.454 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:46.454 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.278144Z", 2020-07-17T12:06:46.454 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:46.454 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:46.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:46.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:46.455 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:46.455 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:46.455 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:46.456 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:46.456 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:46.456 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:46.456 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:46.456 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:06:46.456 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:46.457 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:46.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:46.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:46.457 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:46.457 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:46.458 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:46.458 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:46.458 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:46.458 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:46.458 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:46.459 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:46.459 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:46.459 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:46.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:46.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:46.459 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:46.460 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:46.460 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:46.460 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:46.460 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:46.460 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:46.461 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:46.461 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:46.461 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:46.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:46.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:46.461 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:46.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:46.462 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:46.462 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:46.462 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:46.462 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:46.463 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:46.464 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:06:46.464 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:06:46.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:46.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:46.466 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:46.506 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:46.507 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:46.744 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:06:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:06:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:06:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:06:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987605.278144_59.1", 2020-07-17T12:06:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:06:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.278144Z", 2020-07-17T12:06:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:06:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:06:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:06:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:06:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:46.767 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:46.769 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:06:46.770 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:06:46.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:46.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:46.772 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:46.814 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:46.815 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:47.092 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:47.092 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:47.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.094 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:47.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.094 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:47.094 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987605.278144_59.1", 2020-07-17T12:06:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.278144Z", 2020-07-17T12:06:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:47.098 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:47.098 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:06:47.098 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:47.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.099 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:47.099 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.099 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:47.099 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:47.099 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:47.100 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:47.100 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:47.100 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:47.100 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:47.100 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.101 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:47.102 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.102 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:47.102 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:47.102 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:47.103 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:47.103 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:47.103 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:47.103 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:47.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.104 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:47.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.104 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:47.104 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:47.104 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:47.105 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.105 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:47.106 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:06:47.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:47.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:47.108 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:47.147 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:47.148 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:47.360 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:47.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:47.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.363 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:47.363 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.5.5" 2020-07-17T12:06:47.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.364 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:47.365 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:47.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.7.5" 2020-07-17T12:06:47.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:06:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:47.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:06:47.368 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.369 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:06:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:06:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.5.5" 2020-07-17T12:06:47.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.372 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:06:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.4.5" 2020-07-17T12:06:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.374 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:06:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:47.375 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:06:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:47.376 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:47.377 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:47.378 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-10 2020-07-17T12:06:47.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:47.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:47.381 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:47.437 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:47.437 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:47.642 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:06:47.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.643 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:47.643 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.643 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.645 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:47.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.645 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.645 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:47.646 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:47.646 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.646 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.647 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:47.647 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.647 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:47.647 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.647 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.648 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:06:47.648 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.648 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.648 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:06:47.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.649 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.649 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.5.5", 2020-07-17T12:06:47.649 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.463829Z" 2020-07-17T12:06:47.649 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.649 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.650 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:47.650 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.650 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:47.651 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.651 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.651 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:47.651 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.652 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.652 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:47.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.652 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.652 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:47.653 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:47.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.654 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:47.654 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.654 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:47.654 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.654 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.655 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "_myobj", 2020-07-17T12:06:47.655 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.655 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.655 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:06:47.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.656 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.656 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.7.5", 2020-07-17T12:06:47.657 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.460142Z" 2020-07-17T12:06:47.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.658 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:06:47.658 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.658 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:47.659 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.659 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.659 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:47.659 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.659 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.660 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:47.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.660 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.660 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:47.661 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:47.661 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.661 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:06:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.662 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.663 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:47.663 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.663 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.663 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:47.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.664 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.664 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:47.664 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:47.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.665 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.665 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:06:47.666 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.666 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:47.666 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.666 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.666 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:47.666 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.667 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.667 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:47.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.667 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.668 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:47.668 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:47.668 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.668 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.668 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.669 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:06:47.669 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.669 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:47.669 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.670 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.670 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "&", 2020-07-17T12:06:47.670 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.670 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.671 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:06:47.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.671 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.671 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.5.5", 2020-07-17T12:06:47.671 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.695757Z" 2020-07-17T12:06:47.672 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.672 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.672 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.672 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:06:47.672 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.673 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:47.673 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.673 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.673 INFO:teuthology.orchestra.run.smithi052.stdout: "name": ":", 2020-07-17T12:06:47.673 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.674 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.674 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:06:47.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.675 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.675 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.4.5", 2020-07-17T12:06:47.675 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.656596Z" 2020-07-17T12:06:47.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.677 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:06:47.677 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.677 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:47.677 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.678 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.678 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:47.679 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.679 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.679 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:47.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.679 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.680 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:47.680 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:47.680 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:47.681 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:06:47.682 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:47.682 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:47.682 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:47.683 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:47.683 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:47.683 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:47.684 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:47.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.684 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:47.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:47.685 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:47.685 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:47.685 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:47.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:47.687 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:06:47.687 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-10 2020-07-17T12:06:47.688 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-10 zones={test-zone2, test-zone1} 2020-07-17T12:06:47.701 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:06:47.728 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:47.729 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:47.730 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:47.730 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:47.731 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:06:47.738 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:47.739 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:47.739 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:47.740 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:47.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:47.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:47.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:48.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:48.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:48.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:48.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:48.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:48.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:48.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:48.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:48.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:48.878 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-10 zones={test-zone2, test-zone1} 2020-07-17T12:06:48.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:48.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:48.881 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:48.922 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:48.922 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:49.146 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:49.147 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:49.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.147 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:49.147 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:49.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.148 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:49.148 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.5.5" 2020-07-17T12:06:49.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.149 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:49.149 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:49.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.149 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:49.150 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.7.5" 2020-07-17T12:06:49.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.150 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:06:49.150 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:49.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.151 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:06:49.151 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:49.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.152 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:06:49.152 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:49.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.152 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:06:49.153 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.5.5" 2020-07-17T12:06:49.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.153 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:06:49.153 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.4.5" 2020-07-17T12:06:49.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.154 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:06:49.154 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:49.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.155 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:06:49.155 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:49.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.156 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:49.156 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:49.157 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-10 2020-07-17T12:06:49.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:49.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:49.160 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:49.421 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:49.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.422 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:49.422 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.423 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:49.423 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.423 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.423 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:49.423 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.424 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.424 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:49.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.424 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.425 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:49.425 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:49.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.426 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:49.426 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.426 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:49.427 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.427 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.427 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:06:49.428 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.428 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.429 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:49.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.430 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.430 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.5.5", 2020-07-17T12:06:49.431 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.457996Z" 2020-07-17T12:06:49.431 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.431 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.432 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:49.432 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.433 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:49.433 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.433 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.434 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:49.434 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.434 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.434 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:49.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.435 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.435 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:49.435 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:49.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.436 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:49.436 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.437 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:49.437 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.437 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.437 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "_myobj", 2020-07-17T12:06:49.437 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.438 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.438 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:49.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.438 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.439 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.7.5", 2020-07-17T12:06:49.439 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.458036Z" 2020-07-17T12:06:49.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.440 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:06:49.440 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.440 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:49.440 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.440 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.441 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:49.441 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.441 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.441 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:49.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.442 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.442 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:49.442 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:49.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.443 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:06:49.443 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.444 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:49.444 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.444 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.444 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:49.445 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.445 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.446 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:49.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.446 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.447 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:49.447 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:49.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.522 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:06:49.523 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.523 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:49.523 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.524 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.524 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:49.524 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.525 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.525 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:49.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.526 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.526 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:49.527 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:49.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.528 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:06:49.529 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.529 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:49.529 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.530 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.531 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "&", 2020-07-17T12:06:49.531 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.531 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.532 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:49.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.532 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.532 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.5.5", 2020-07-17T12:06:49.533 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.657978Z" 2020-07-17T12:06:49.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.534 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:06:49.534 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.534 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:49.535 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.535 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.535 INFO:teuthology.orchestra.run.smithi146.stdout: "name": ":", 2020-07-17T12:06:49.535 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.535 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.536 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:49.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.536 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.536 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.4.5", 2020-07-17T12:06:49.536 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.654009Z" 2020-07-17T12:06:49.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.537 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:06:49.538 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.538 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:49.538 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.539 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.539 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:49.540 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.540 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.540 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:49.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.541 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.541 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:49.541 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:49.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:49.542 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:06:49.542 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:49.542 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:49.543 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:49.543 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:49.543 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:49.543 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:49.543 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:49.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.544 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:49.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:49.544 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:49.548 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:49.548 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:49.548 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.548 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:49.549 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-10 2020-07-17T12:06:49.550 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-10 zones={test-zone3, test-zone1} 2020-07-17T12:06:49.574 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:06:49.582 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:49.582 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:49.583 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:49.584 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:49.585 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:06:49.599 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:49.600 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:49.601 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:49.602 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:49.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:49.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:49.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:49.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:50.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:50.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:50.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:50.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:50.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:50.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:50.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:50.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:50.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-10 zones={test-zone3, test-zone1} 2020-07-17T12:06:50.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:50.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:50.800 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:50.840 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:50.841 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:51.063 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:51.063 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:51.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.5.5" 2020-07-17T12:06:51.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.7.5" 2020-07-17T12:06:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:06:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.069 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:06:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:06:51.070 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:06:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.5.5" 2020-07-17T12:06:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:06:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.4.5" 2020-07-17T12:06:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.072 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:06:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:06:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:51.075 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:51.076 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-10 2020-07-17T12:06:51.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:51.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:51.079 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:51.126 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:51.127 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:51.343 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:06:51.343 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.343 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:51.344 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.344 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:51.344 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.344 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.344 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.345 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.345 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.345 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.345 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.346 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:51.346 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:51.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.347 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.347 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:51.347 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.347 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:51.347 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.348 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.348 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.348 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.348 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.349 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.349 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.350 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.5.5", 2020-07-17T12:06:51.350 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.302875Z" 2020-07-17T12:06:51.350 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.350 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.351 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:51.351 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.351 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:51.351 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.352 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.352 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.352 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.352 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.352 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.353 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.353 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:51.353 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:51.355 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.356 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.356 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:51.356 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.357 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:51.357 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.357 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.357 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.357 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.358 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.358 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.358 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.359 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.7.5", 2020-07-17T12:06:51.359 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.357162Z" 2020-07-17T12:06:51.359 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.359 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.360 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:06:51.360 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.360 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:51.360 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.360 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.361 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.361 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.361 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.361 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.362 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.362 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:51.362 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:51.362 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.362 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.365 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:06:51.365 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.365 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:51.365 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.366 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.366 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.366 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.366 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.367 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.367 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.367 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:51.368 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:51.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.369 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.369 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:06:51.370 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.370 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:51.370 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.370 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.371 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.371 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.371 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.372 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.372 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.373 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:51.373 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:51.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.374 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.374 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:06:51.374 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.374 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:51.375 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.375 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.375 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.375 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.376 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.376 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.376 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.377 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.5.5", 2020-07-17T12:06:51.377 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.460280Z" 2020-07-17T12:06:51.377 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.377 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.377 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.378 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:06:51.378 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.378 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:51.378 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.378 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.381 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.381 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.382 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.382 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.383 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.383 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.4.5", 2020-07-17T12:06:51.384 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.406670Z" 2020-07-17T12:06:51.384 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.384 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.384 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:06:51.385 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.385 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:51.385 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.385 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.401 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.401 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.401 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.402 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.402 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.403 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:51.403 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:51.403 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.403 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.404 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.404 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:06:51.404 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:51.405 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:51.405 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:51.405 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:51.406 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:51.406 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:51.406 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:51.406 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.407 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:51.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.408 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:51.408 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:51.408 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:51.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.410 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.410 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:06:51.411 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-10 2020-07-17T12:06:51.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:51.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:51.413 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:51.493 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:51.494 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:51.713 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:51.714 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:51.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.724 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:51.724 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:51.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.724 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.724 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:51.725 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.9.5" 2020-07-17T12:06:51.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.725 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:51.725 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:51.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.731 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:51.731 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.10.5" 2020-07-17T12:06:51.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:06:51.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:51.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:06:51.741 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:51.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.741 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:06:51.741 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:51.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.742 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:06:51.742 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.4.5" 2020-07-17T12:06:51.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.743 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:06:51.743 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.7.5" 2020-07-17T12:06:51.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.744 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:06:51.744 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:51.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:51.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:51.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:06:51.745 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:51.745 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:51.745 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:51.746 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:51.747 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-11 2020-07-17T12:06:51.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:51.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:51.748 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:51.796 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:51.797 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:52.020 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:52.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.021 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:52.021 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.021 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:52.022 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.022 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.022 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:52.023 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.023 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.023 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:52.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.024 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.024 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:52.025 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:52.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.027 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:52.027 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.027 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:52.028 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.028 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.028 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:06:52.028 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.029 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.029 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:52.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.030 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.031 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.9.5", 2020-07-17T12:06:52.031 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.706183Z" 2020-07-17T12:06:52.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.033 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:52.033 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.033 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:52.033 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.034 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.034 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:52.034 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.034 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.035 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:52.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.035 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.035 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:52.035 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:52.036 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.036 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.036 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:52.037 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.037 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:52.037 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.037 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.037 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "_myobj", 2020-07-17T12:06:52.038 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.038 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.038 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:52.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.039 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.039 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.10.5", 2020-07-17T12:06:52.039 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.706209Z" 2020-07-17T12:06:52.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.040 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:06:52.040 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.040 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:52.040 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.041 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.041 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:52.041 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.042 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.042 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:52.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.043 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.043 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:52.043 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:52.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.045 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:06:52.045 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.046 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:52.046 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.046 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.047 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:52.047 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.047 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.048 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:52.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.049 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.049 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:52.050 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:52.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.052 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:06:52.053 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.053 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:52.053 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.054 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.054 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:52.054 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.054 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.055 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:52.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.056 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.056 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:52.056 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:52.057 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.057 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.058 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:06:52.058 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.059 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:52.059 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.059 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.060 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "&", 2020-07-17T12:06:52.060 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.060 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.061 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:52.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.062 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.062 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.4.5", 2020-07-17T12:06:52.062 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.913998Z" 2020-07-17T12:06:52.063 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.063 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.063 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:06:52.064 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.064 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:52.064 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.064 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.064 INFO:teuthology.orchestra.run.smithi146.stdout: "name": ":", 2020-07-17T12:06:52.065 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.065 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.066 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:52.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.067 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.067 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.7.5", 2020-07-17T12:06:52.067 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.662434Z" 2020-07-17T12:06:52.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.068 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:06:52.068 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.068 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:52.068 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.070 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.070 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:52.070 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.071 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.071 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:52.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.072 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.072 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:52.073 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:52.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:52.075 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:06:52.075 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:52.075 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:52.076 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:52.076 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:52.076 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:52.077 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:52.077 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:52.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.078 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:52.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:52.078 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:52.079 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:52.079 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:52.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:52.080 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:52.081 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-11 2020-07-17T12:06:52.082 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-11 zones={test-zone1, test-zone2} 2020-07-17T12:06:52.099 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:06:52.107 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:52.108 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:52.108 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:52.109 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:52.110 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:06:52.116 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:52.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:52.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:52.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:52.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:52.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:52.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:52.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:52.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:52.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:53.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:53.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:53.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:53.298 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-11 zones={test-zone1, test-zone2} 2020-07-17T12:06:53.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:53.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:53.301 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:53.341 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:53.341 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:53.570 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:53.571 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:53.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.571 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:53.571 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:53.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.572 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:53.572 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.9.5" 2020-07-17T12:06:53.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.573 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:53.573 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:53.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.573 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:53.574 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.10.5" 2020-07-17T12:06:53.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.574 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:06:53.575 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:53.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.576 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:06:53.576 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:53.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.576 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:06:53.577 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:53.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.577 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:06:53.579 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.4.5" 2020-07-17T12:06:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:06:53.580 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.7.5" 2020-07-17T12:06:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.581 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:06:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:53.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:06:53.583 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:06:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:53.585 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:53.586 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:53.587 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-11 2020-07-17T12:06:53.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:53.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:53.590 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:53.647 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:53.648 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:53.854 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:53.862 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.862 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:06:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.9.5", 2020-07-17T12:06:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.718111Z" 2020-07-17T12:06:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "_myobj", 2020-07-17T12:06:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.10.5", 2020-07-17T12:06:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.677979Z" 2020-07-17T12:06:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:06:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:06:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:06:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:06:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "&", 2020-07-17T12:06:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.4.5", 2020-07-17T12:06:53.906 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.729967Z" 2020-07-17T12:06:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:06:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "name": ":", 2020-07-17T12:06:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.7.5", 2020-07-17T12:06:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.718066Z" 2020-07-17T12:06:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:06:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:06:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:53.919 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:53.919 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-11 2020-07-17T12:06:53.920 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-11 zones={test-zone3, test-zone2} 2020-07-17T12:06:53.932 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:06:53.939 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:53.939 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:53.940 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:53.940 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:53.941 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:06:53.948 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:53.948 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:53.948 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:53.949 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:53.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:54.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:54.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:54.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:54.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:54.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:54.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:54.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:54.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:54.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:54.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:55.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:55.105 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-11 zones={test-zone3, test-zone2} 2020-07-17T12:06:55.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:55.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:55.107 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:55.149 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:55.150 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:55.366 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:55.366 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:55.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.367 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:55.367 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:55.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.368 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:55.368 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.7.5" 2020-07-17T12:06:55.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.368 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:55.369 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:55.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.369 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:55.369 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.7.5" 2020-07-17T12:06:55.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.370 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:06:55.370 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:55.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.371 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:06:55.371 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:55.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.371 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:06:55.371 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:55.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.372 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:06:55.372 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.12.5" 2020-07-17T12:06:55.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.373 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:06:55.373 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.4.5" 2020-07-17T12:06:55.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.374 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:06:55.374 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:55.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.374 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:06:55.375 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:55.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.375 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:55.375 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:55.376 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-12 2020-07-17T12:06:55.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:55.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:55.380 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:55.459 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:55.459 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:55.670 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:55.673 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.673 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.674 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:55.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.674 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.674 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:55.674 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:55.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.675 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:55.676 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.676 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:55.676 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.676 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.676 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:06:55.676 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.677 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.677 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:55.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.677 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.678 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.7.5", 2020-07-17T12:06:55.678 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:46.005893Z" 2020-07-17T12:06:55.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.679 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:55.679 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.686 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:55.686 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.686 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.686 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:55.687 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.687 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.687 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:55.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.687 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.688 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:55.688 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:55.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.690 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:55.691 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.691 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:55.691 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.692 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.692 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "_myobj", 2020-07-17T12:06:55.692 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.693 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:55.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.694 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.694 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.7.5", 2020-07-17T12:06:55.695 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:46.001880Z" 2020-07-17T12:06:55.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.696 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:06:55.696 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.696 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:55.696 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.697 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.697 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:55.697 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.698 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:55.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.698 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.699 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:55.699 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:55.699 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.701 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:06:55.701 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.701 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:55.702 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.702 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.702 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:55.703 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.703 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.703 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:55.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.704 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.704 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:55.704 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:55.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.705 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:06:55.705 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.706 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:55.706 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.706 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.706 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:55.707 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.707 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.708 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:55.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.708 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.708 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:55.709 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:55.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.709 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:06:55.710 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.710 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:55.710 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.710 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.710 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "&", 2020-07-17T12:06:55.711 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.711 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.711 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:55.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.712 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.712 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.12.5", 2020-07-17T12:06:55.712 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.965174Z" 2020-07-17T12:06:55.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.714 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:06:55.715 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.715 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:06:55.715 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.715 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.715 INFO:teuthology.orchestra.run.smithi146.stdout: "name": ":", 2020-07-17T12:06:55.715 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.716 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.717 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:06:55.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.717 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.718 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.4.5", 2020-07-17T12:06:55.718 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.966016Z" 2020-07-17T12:06:55.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.719 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:06:55.719 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.719 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:55.719 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.720 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.720 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:55.721 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.721 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.721 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:55.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.722 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.722 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:55.722 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:55.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:55.723 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:06:55.723 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:55.724 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:06:55.724 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:06:55.724 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:06:55.724 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:06:55.725 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:06:55.725 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:06:55.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.725 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:06:55.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:55.726 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:06:55.726 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:06:55.727 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:06:55.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:55.728 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:55.729 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-12 2020-07-17T12:06:55.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-12 zones={test-zone1, test-zone3} 2020-07-17T12:06:55.742 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:06:55.748 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:55.749 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:55.749 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:55.750 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:55.751 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:06:55.755 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:55.756 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:55.756 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:55.756 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:55.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:55.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:55.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:56.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:56.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:56.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:56.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:56.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:56.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:56.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:56.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:56.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:56.889 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-12 zones={test-zone1, test-zone3} 2020-07-17T12:06:56.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:56.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:56.891 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:56.928 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:56.929 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:57.137 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.137 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.7.5" 2020-07-17T12:06:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:57.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.7.5" 2020-07-17T12:06:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:06:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.141 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:06:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:06:57.142 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:06:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.12.5" 2020-07-17T12:06:57.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:06:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.4.5" 2020-07-17T12:06:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.144 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:06:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:06:57.145 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:06:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:57.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:57.146 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:57.147 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-12 2020-07-17T12:06:57.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:57.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:57.149 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:57.195 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:57.196 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:57.405 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:06:57.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.409 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:57.409 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.410 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:57.411 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.411 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.411 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:06:57.411 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.411 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.412 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:06:57.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.412 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.412 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.7.5", 2020-07-17T12:06:57.412 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.959406Z" 2020-07-17T12:06:57.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.413 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:57.414 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.414 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:57.414 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.414 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.414 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:57.415 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.415 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.415 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:57.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.415 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.416 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:57.416 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:57.416 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.416 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.417 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:57.417 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.417 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:57.417 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.417 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.418 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "_myobj", 2020-07-17T12:06:57.418 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.418 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.418 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:06:57.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.419 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.419 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.7.5", 2020-07-17T12:06:57.419 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.999808Z" 2020-07-17T12:06:57.419 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.419 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.420 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.421 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:06:57.421 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.421 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:57.421 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.421 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.422 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:57.422 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.422 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.422 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:57.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.423 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.423 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:57.423 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:57.423 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.423 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.424 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:06:57.424 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.424 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:57.424 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.425 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.425 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:57.425 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.425 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.426 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:57.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.426 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.426 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:57.426 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:57.426 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.427 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.427 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:06:57.427 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.428 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:57.428 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.428 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.428 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:57.428 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.428 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.429 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:57.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.429 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.429 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:57.430 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:57.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.431 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:06:57.431 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.431 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:57.432 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.432 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.432 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "&", 2020-07-17T12:06:57.432 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.432 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.433 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:06:57.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.433 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.433 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.12.5", 2020-07-17T12:06:57.433 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.999841Z" 2020-07-17T12:06:57.434 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.434 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.434 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.434 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:06:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:06:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: "name": ":", 2020-07-17T12:06:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:06:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.4.5", 2020-07-17T12:06:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.920781Z" 2020-07-17T12:06:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:06:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.439 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.439 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.441 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.441 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:57.441 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:06:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:06:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:06:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:06:57.446 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:06:57.446 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:06:57.446 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:06:57.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.446 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:06:57.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:57.447 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:06:57.447 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:06:57.447 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:06:57.447 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.447 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.448 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:57.448 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:06:57.448 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-12 2020-07-17T12:06:57.449 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-12 zones={test-zone2, test-zone3} 2020-07-17T12:06:57.480 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:06:57.488 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:57.488 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:57.489 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:57.489 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:57.490 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:06:57.499 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-17T12:06:57.500 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-17T12:06:57.501 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-17T12:06:57.501 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:06:57.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:57.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:57.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-17T12:06:57.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:57.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:57.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-17T12:06:58.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:58.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:58.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-17T12:06:58.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:58.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:58.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:06:58.694 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-12 zones={test-zone2, test-zone3} 2020-07-17T12:06:58.695 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-07-17T12:06:58.697 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-13 2020-07-17T12:06:58.735 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-14 2020-07-17T12:06:58.790 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-15 2020-07-17T12:06:59.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:59.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:59.004 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:59.044 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:59.045 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:59.216 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:06:59.266 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:06:59.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:59.267 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987618.830894_71.1", 2020-07-17T12:06:59.268 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:58.830894Z" 2020-07-17T12:06:59.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:59.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:59.269 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:59.269 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:06:59.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:59.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:59.270 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:59.270 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:59.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:59.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:59.271 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:59.272 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:06:59.272 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:59.272 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:06:59.274 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:06:59.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:59.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:59.277 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:59.321 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:59.322 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:59.506 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:06:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:06:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:06:59.507 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:06:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:06:59.508 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:59.509 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:06:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:06:59.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987618.830894_71.1", 2020-07-17T12:06:59.511 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:06:59.512 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.830894Z", 2020-07-17T12:06:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:59.513 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:59.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:06:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:59.515 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:59.516 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:59.517 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:06:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:59.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:06:59.519 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:59.520 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:59.521 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:59.523 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:59.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:06:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:06:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:06:59.525 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:06:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:06:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:06:59.526 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:06:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:06:59.527 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:06:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:06:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:59.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:06:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:06:59.529 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:06:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:06:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:06:59.530 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:06:59.531 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:06:59.533 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:06:59.534 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:06:59.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:59.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:59.537 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:59.577 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:59.578 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:06:59.781 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:06:59.782 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:06:59.782 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:06:59.782 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:06:59.782 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:06:59.783 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:06:59.783 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:59.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:59.783 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:06:59.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:59.784 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:06:59.784 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:59.784 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:59.785 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987618.830894_71.1", 2020-07-17T12:06:59.785 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:59.785 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:06:59.785 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:59.785 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.830894Z", 2020-07-17T12:06:59.786 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:59.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:59.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:59.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:59.787 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:06:59.787 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:59.787 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:59.787 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:06:59.787 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:59.788 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:59.788 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:59.788 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:06:59.788 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:59.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:59.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:59.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:59.789 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:06:59.789 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:59.789 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:59.790 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:59.790 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:59.790 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:59.790 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:59.791 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:59.791 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:59.791 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:59.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:59.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:06:59.792 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:06:59.792 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:06:59.792 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:06:59.792 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:06:59.793 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:06:59.793 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:06:59.793 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:06:59.793 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:06:59.793 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:06:59.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:59.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:59.794 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:06:59.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:06:59.796 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:06:59.796 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:06:59.797 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:06:59.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:06:59.799 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:06:59.800 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:06:59.801 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:06:59.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:59.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:06:59.804 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:06:59.843 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:06:59.844 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:00.114 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:07:00.114 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:07:00.115 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:07:00.115 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:07:00.115 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:07:00.115 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:07:00.116 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:00.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.116 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:07:00.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.117 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:00.117 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.117 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:00.117 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987618.830894_71.1", 2020-07-17T12:07:00.118 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:00.118 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:07:00.118 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:00.118 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.830894Z", 2020-07-17T12:07:00.118 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:00.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.119 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:00.119 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.120 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:00.120 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:07:00.120 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:00.120 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:00.121 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:00.121 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:07:00.121 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:00.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.139 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:00.139 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.139 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:00.139 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:07:00.140 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:00.140 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:00.140 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:00.140 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:07:00.141 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:00.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.142 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:00.142 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.142 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:00.142 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:07:00.143 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:00.143 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:00.143 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:00.143 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:07:00.144 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:00.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.144 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:07:00.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.145 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:07:00.145 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:07:00.145 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:07:00.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.146 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:07:00.147 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:07:00.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:00.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:00.149 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:00.194 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:00.195 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:00.389 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:00.390 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:00.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.390 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:00.390 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.11.5" 2020-07-17T12:07:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.393 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:00.393 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.394 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:00.394 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.394 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:00.395 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.395 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:00.396 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.396 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:00.396 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.403 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:00.403 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.404 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:00.405 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:00.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:00.407 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:00.407 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:00.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:00.407 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:00.408 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:00.409 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:00.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:00.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:00.412 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:00.457 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:00.457 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:00.651 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:00.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.652 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:00.653 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.653 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.654 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.654 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.654 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.654 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.655 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.655 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.656 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.656 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.656 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.657 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:00.658 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.658 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:00.658 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.658 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.658 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:07:00.659 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.659 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.659 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:07:00.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.660 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.661 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.11.5", 2020-07-17T12:07:00.661 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:59.064046Z" 2020-07-17T12:07:00.661 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.662 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:00.663 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.663 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.663 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.663 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.664 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.664 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.664 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.665 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.665 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.666 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.666 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.666 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.666 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.667 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.667 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:00.667 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.668 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.668 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.669 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.669 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.670 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.670 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.671 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.672 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.672 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.673 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.674 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.675 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:00.675 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.676 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.676 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.676 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.677 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.677 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.678 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.679 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.680 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.680 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.681 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.683 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:00.683 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.684 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.684 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.684 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.685 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.685 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.686 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.686 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.687 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.687 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.688 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.689 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.690 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:00.690 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.690 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.691 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.692 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.692 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.692 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.693 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.694 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.694 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.695 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.695 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.697 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:00.697 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.698 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.698 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.698 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.699 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.699 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.699 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.700 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.701 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.702 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.702 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.702 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.703 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.704 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:00.704 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.705 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.705 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.705 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.706 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.706 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.706 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.707 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.708 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.708 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.709 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.711 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:00.712 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.712 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.713 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.713 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.714 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.714 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.715 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.718 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.719 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.720 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.720 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.720 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.721 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:00.722 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:00.723 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:00.723 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:00.723 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:00.724 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:00.724 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:00.725 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:00.725 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:00.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.726 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:00.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:00.727 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:00.727 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:00.728 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:00.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.729 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:00.729 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:00.730 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:00.731 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-13 zones={test-zone2, test-zone1} 2020-07-17T12:07:00.750 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:00.806 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:00.808 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:00.816 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:00.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:00.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:01.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:01.314 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-13 zones={test-zone2, test-zone1} 2020-07-17T12:07:01.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:01.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:01.317 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:01.363 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:01.363 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:01.548 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:01.549 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.11.5" 2020-07-17T12:07:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:01.557 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:01.558 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:01.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:01.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:01.560 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:01.643 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:01.643 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:01.847 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:01.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.848 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:01.848 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.848 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.848 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.849 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.849 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.849 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.849 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.850 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.850 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.850 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.851 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.852 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:01.852 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.852 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:01.852 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.853 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.853 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:07:01.853 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.853 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.854 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:01.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.854 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.854 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.11.5", 2020-07-17T12:07:01.855 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:59.061970Z" 2020-07-17T12:07:01.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.856 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:01.856 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.856 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.856 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.857 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.857 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.857 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.857 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.858 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.858 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.858 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.859 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.860 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:01.860 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.860 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.860 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.860 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.861 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.861 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.861 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.862 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.862 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.862 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.863 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.864 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:01.864 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.864 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.865 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.865 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.865 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.866 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.866 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.866 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.867 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:01.895 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:01.896 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:01.896 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-13 zones={test-zone3, test-zone1} 2020-07-17T12:07:01.910 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:01.958 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:01.959 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:01.972 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:02.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:02.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:02.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:02.319 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-13 zones={test-zone3, test-zone1} 2020-07-17T12:07:02.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:02.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:02.321 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:02.358 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:02.359 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:02.570 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:02.571 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:02.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.572 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:02.572 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.573 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:02.573 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.11.5" 2020-07-17T12:07:02.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.574 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:02.574 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.575 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:02.575 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.576 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:02.577 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.578 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:02.578 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.579 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:02.579 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.580 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:02.580 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.581 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:02.582 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.583 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:02.583 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:02.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:02.584 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:02.584 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:02.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:02.584 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:02.585 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:02.586 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:02.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:02.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:02.588 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:02.632 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:02.633 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:02.849 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:02.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.850 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:02.850 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.850 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.850 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.851 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.851 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.851 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.851 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.852 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.852 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.852 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.852 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.852 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.853 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:02.853 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.854 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:02.854 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.854 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.854 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.854 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.855 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.855 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.855 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.855 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.11.5", 2020-07-17T12:07:02.856 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.856094Z" 2020-07-17T12:07:02.856 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.856 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.857 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:02.857 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.857 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.857 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.857 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.857 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.858 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.858 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.858 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.859 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.859 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.859 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.859 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.859 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.860 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:02.860 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.860 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.860 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.861 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.861 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.861 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.861 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.862 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.862 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.862 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.862 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.863 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.863 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:02.863 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.864 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.864 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.864 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.864 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.864 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.865 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.865 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.865 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.866 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.866 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.866 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.866 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.867 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:02.867 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.867 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.867 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.867 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.868 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.868 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.868 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.868 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.869 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.869 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.869 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.869 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.869 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.870 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:02.870 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.870 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.870 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.871 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.871 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.871 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.871 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.872 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.872 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.872 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.872 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.872 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.873 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.873 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.873 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:02.873 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.874 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.874 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.874 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.874 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.874 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.874 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.875 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.875 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.875 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.876 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.876 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.876 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.876 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:02.877 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.877 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.877 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.877 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.877 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.878 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.878 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.878 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.878 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.879 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.879 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.879 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.879 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.880 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:02.880 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.880 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.880 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.880 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.881 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.881 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.881 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.881 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.910 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.910 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.910 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.911 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.911 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:02.911 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:02.912 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:02.912 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:02.912 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:02.912 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:02.912 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:02.912 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:02.913 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:02.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.913 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:02.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:02.913 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:02.914 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:02.914 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:02.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:02.914 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:02.915 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:02.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:02.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:02.917 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:03.007 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:03.007 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:03.219 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:07:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:07:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.12.5" 2020-07-17T12:07:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:07:03.228 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:07:03.229 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-14 2020-07-17T12:07:03.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:03.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:03.230 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:03.272 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:03.273 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:03.457 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:03.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.458 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:03.458 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.458 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.459 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.459 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.459 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.460 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.460 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.461 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.461 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.462 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.462 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.463 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.464 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:03.464 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.464 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:03.465 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.465 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.465 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:07:03.466 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.466 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.467 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:03.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.467 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.468 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.12.5", 2020-07-17T12:07:03.468 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:59.125907Z" 2020-07-17T12:07:03.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.469 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.470 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:03.470 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.471 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.471 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.472 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.472 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.472 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.473 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.473 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.474 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.474 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.474 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.477 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:03.477 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.478 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.478 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.478 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.479 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.479 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.479 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.482 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.482 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.483 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.483 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.485 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:03.485 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.486 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.486 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.486 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.487 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.487 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.487 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.488 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.488 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.488 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.489 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.489 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.489 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.490 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:03.490 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.490 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.490 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.490 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.491 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.491 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.491 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.492 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.492 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.493 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.493 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.494 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:03.494 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.495 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.495 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.495 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.495 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.495 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.496 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.496 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.496 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.497 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.497 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.498 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:03.498 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.498 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.499 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.499 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.499 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.499 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.500 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.500 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.500 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.501 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.501 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.503 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:03.503 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.503 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.503 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.503 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.504 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.504 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.504 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.505 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.505 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.505 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.505 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.506 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:03.507 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.507 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.507 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.507 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.507 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.508 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.508 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.510 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.511 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.511 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.511 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:03.512 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:03.512 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:03.513 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:03.513 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:03.513 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:03.514 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:03.514 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:03.514 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:03.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.514 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:03.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:03.515 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:03.515 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:03.515 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:03.515 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:03.516 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:03.516 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-14 2020-07-17T12:07:03.518 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-14 zones={test-zone1, test-zone2} 2020-07-17T12:07:03.532 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:03.582 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:03.583 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:03.591 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:03.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:03.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:03.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:03.938 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-14 zones={test-zone1, test-zone2} 2020-07-17T12:07:03.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:03.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:03.941 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:03.982 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:03.982 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:04.200 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:07:04.201 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:07:04.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.201 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:04.202 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.203 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:04.203 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.12.5" 2020-07-17T12:07:04.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.204 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:04.204 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.205 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:04.206 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.207 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:04.207 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.208 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:04.208 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.209 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:04.210 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.211 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:04.211 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.213 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:04.213 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.214 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:04.214 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:04.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:04.215 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:04.215 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:04.216 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:04.216 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:07:04.216 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:07:04.218 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-14 2020-07-17T12:07:04.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:04.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:04.220 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:04.455 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:04.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.455 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:04.456 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.456 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.456 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.456 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.456 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.457 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.457 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.457 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.458 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.458 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.458 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.459 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:04.460 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.460 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:04.460 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.460 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.460 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:07:04.461 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.461 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.461 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:04.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.462 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.462 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.12.5", 2020-07-17T12:07:04.462 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:59.122019Z" 2020-07-17T12:07:04.463 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.463 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.464 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:04.464 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.464 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.465 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.465 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.465 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.466 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.466 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.467 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.467 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.468 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.468 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.469 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:04.470 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.470 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.470 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.471 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.471 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.471 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.472 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.472 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.473 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.473 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.473 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.475 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:04.475 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.476 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.476 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.476 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.477 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.477 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.477 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.478 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.479 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.479 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.479 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.481 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:04.481 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.482 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.482 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.482 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.482 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.483 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.483 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.484 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.484 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.485 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.485 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.486 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.487 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:04.488 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.489 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.489 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.489 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.490 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.490 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.490 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.491 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.492 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.492 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.492 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.494 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:04.494 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.495 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.495 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.495 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.496 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.496 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.496 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.497 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.498 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.498 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.498 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.500 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:04.500 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.500 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.501 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.501 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.501 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.502 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.502 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.502 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.503 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.503 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.504 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.505 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.507 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:04.509 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:04.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:04.511 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:04.512 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:04.513 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-14 2020-07-17T12:07:04.514 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-14 zones={test-zone3, test-zone2} 2020-07-17T12:07:04.526 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:04.579 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:04.580 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:04.588 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:04.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:04.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:04.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:04.926 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-14 zones={test-zone3, test-zone2} 2020-07-17T12:07:04.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:04.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:04.928 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:05.180 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:05.180 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:05.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.181 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:05.181 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.182 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:05.182 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.7.5" 2020-07-17T12:07:05.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.183 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:05.183 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.184 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:05.184 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.185 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:05.185 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.186 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:05.186 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.186 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:05.186 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.187 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:05.187 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.188 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:05.188 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.188 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:05.189 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.189 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:05.189 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:05.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.190 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:05.190 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:05.191 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-15 2020-07-17T12:07:05.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:05.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:05.193 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:05.293 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:05.293 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:05.468 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:07:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.7.5", 2020-07-17T12:07:05.480 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:59.166070Z" 2020-07-17T12:07:05.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.481 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:05.481 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.487 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.488 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.488 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.488 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.489 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.489 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.490 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.490 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.491 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.491 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.491 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.493 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:05.493 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.493 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.494 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.494 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.494 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.495 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.495 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.495 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.496 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.496 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.497 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.498 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:05.499 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.499 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.499 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.500 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.500 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.500 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.501 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.502 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.503 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.503 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.504 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.505 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:05.506 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.506 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.506 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.507 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.507 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.507 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.508 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.508 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.509 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.509 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.510 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.511 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:05.512 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.512 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.512 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.513 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.513 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.513 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.514 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.514 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.515 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.515 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.515 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.517 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:05.517 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.518 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.518 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.518 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.519 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.519 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.519 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.520 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.520 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.521 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.521 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.523 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:05.523 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.523 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.524 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.524 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.524 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.525 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.525 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.525 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.526 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.526 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.526 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.528 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:05.528 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.528 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.529 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.529 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.529 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.529 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.530 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.531 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.532 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.532 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.532 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:05.533 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:05.534 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:05.534 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:05.534 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:05.534 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:05.535 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:05.535 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:05.535 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:05.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.535 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:05.537 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:05.538 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-15 2020-07-17T12:07:05.538 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-15 zones={test-zone1, test-zone3} 2020-07-17T12:07:05.551 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:05.598 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:05.599 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:05.606 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:05.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:05.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:05.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:05.932 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-15 zones={test-zone1, test-zone3} 2020-07-17T12:07:05.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:05.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:05.934 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:05.974 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:05.975 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:06.286 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.7.5" 2020-07-17T12:07:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:06.306 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:06.307 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-15 2020-07-17T12:07:06.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:06.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:06.310 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:06.362 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:06.363 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:06.572 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:06.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.573 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:06.574 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.574 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.574 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.575 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.575 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.575 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.576 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.576 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.577 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.577 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.578 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.578 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.578 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.580 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:06.580 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.581 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:06.581 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.583 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.583 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:07:06.584 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.584 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.585 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:07:06.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.585 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.585 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.7.5", 2020-07-17T12:07:06.586 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:59.163916Z" 2020-07-17T12:07:06.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.588 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:06.588 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.588 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.589 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.589 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.590 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.590 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.591 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.591 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.592 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.593 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.593 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.593 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:06.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.595 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.596 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.596 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.596 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.597 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.597 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.598 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.598 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.599 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.599 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.600 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.601 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:06.602 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.602 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.602 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.603 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.603 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.603 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.603 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.604 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.605 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.605 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.605 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.607 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:06.607 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.608 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.608 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.608 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.609 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.609 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.609 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.610 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.611 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.612 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.612 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.613 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.613 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.614 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:06.614 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.615 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.615 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.615 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.616 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.616 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.616 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.617 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.618 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.618 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.618 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.620 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:06.620 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.621 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.621 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.622 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.623 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.623 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.623 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.624 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.625 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.625 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.625 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.625 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.626 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.626 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.627 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:06.627 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.627 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.628 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.628 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.628 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.629 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.629 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.629 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.630 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.630 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.631 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.631 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.631 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.631 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.632 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.633 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:06.633 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.634 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.634 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.634 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.634 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.635 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.635 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.635 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.637 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.637 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.638 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.638 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.638 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.638 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.638 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:06.639 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:06.639 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:06.639 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:06.639 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:06.639 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:06.640 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:06.640 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:06.640 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:06.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.640 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:06.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:06.641 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:06.641 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:06.641 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:06.641 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.641 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:06.642 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:06.642 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-15 2020-07-17T12:07:06.643 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-15 zones={test-zone2, test-zone3} 2020-07-17T12:07:06.655 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:06.706 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:06.707 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:06.714 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-17T12:07:06.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:06.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:06.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-17T12:07:07.040 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-15 zones={test-zone2, test-zone3} 2020-07-17T12:07:07.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:07.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:07.059 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:07.092 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:07.093 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:07.287 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:07.288 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:07.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.288 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:07.289 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.289 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:07.289 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.13.5" 2020-07-17T12:07:07.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.290 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:07.290 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.291 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:07.291 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.293 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:07.294 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.295 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:07.296 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.297 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:07.297 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:07.299 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:07.299 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:07.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:07.300 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:07.300 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:07.300 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:07.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:07.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:07.304 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:07.351 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:07.352 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:07.548 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:07:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:07:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.11.5", 2020-07-17T12:07:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:59.064046Z" 2020-07-17T12:07:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.558 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.558 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.558 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.562 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.568 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.569 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.569 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.570 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:07.571 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.571 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.571 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.572 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.572 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.572 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.572 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.573 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.573 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.573 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.574 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.574 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.577 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.577 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.577 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.584 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:07.584 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.584 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.584 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.588 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.589 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.589 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.589 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.590 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.590 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.591 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.591 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.592 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.592 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.593 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.593 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.601 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.602 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.602 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:07.604 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:07.604 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:07.605 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:07.605 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:07.605 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:07.606 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:07.607 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:07.607 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:07.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.608 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:07.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:07.609 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:07.609 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:07.609 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:07.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:07.611 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:07.611 WARNING:rgw_multi.tests:bucket lbjidp-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.13.5 target=00000000001.11.5 2020-07-17T12:07:12.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:12.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:12.615 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:12.869 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:12.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.870 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:12.870 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.871 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.871 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.871 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.872 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.872 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.872 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.873 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.873 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.874 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.874 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.874 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.874 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.875 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:12.875 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.875 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:12.875 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.875 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.876 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:07:12.876 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.876 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.876 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:07:12.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.877 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.877 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.11.5", 2020-07-17T12:07:12.877 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:59.064046Z" 2020-07-17T12:07:12.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.878 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:12.879 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.879 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.880 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.880 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.880 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.881 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.881 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.881 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.882 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.882 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.882 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.883 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:12.883 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.883 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.884 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.885 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.887 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:12.919 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:12.919 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:12.919 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:12.919 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:12.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:12.921 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:12.922 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:12.923 WARNING:rgw_multi.tests:bucket lbjidp-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.13.5 target=00000000001.11.5 2020-07-17T12:07:17.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:17.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:17.927 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:17.980 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:17.980 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:18.164 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:18.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.164 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:18.164 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.165 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.165 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.165 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.165 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.165 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.166 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.166 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.167 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.167 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.167 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.167 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.168 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:18.168 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.169 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:18.169 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.169 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.169 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:07:18.170 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.170 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.170 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:07:18.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.170 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.171 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.11.5", 2020-07-17T12:07:18.171 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:59.064046Z" 2020-07-17T12:07:18.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.172 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:18.172 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.173 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.173 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.173 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.174 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.174 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.175 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.175 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.175 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.176 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.176 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.176 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.176 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.177 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:18.177 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.177 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.177 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.178 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.178 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.178 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.178 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.178 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.179 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.179 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.179 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.179 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.180 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.180 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:18.180 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.181 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.181 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.181 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.181 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.181 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.182 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.182 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.182 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.182 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.183 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.184 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:18.184 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.184 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.185 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.185 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.185 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.185 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.185 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.186 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.186 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.186 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.187 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.187 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.187 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.187 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:18.188 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.188 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.188 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.188 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.189 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.189 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.189 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.190 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.190 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.191 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.191 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.191 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.191 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.192 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:18.192 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.193 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.193 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.193 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.193 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.193 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.194 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.194 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.194 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.195 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.195 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.195 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.195 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.196 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:18.196 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.196 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.196 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.197 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.197 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.197 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.198 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.199 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.201 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.201 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.201 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.202 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.202 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.203 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.206 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.206 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.207 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.207 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.207 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.207 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:18.208 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:18.208 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:18.208 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:18.208 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:18.208 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:18.209 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:18.209 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:18.209 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:18.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.210 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:18.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:18.210 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:18.210 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:18.211 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:18.211 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.211 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.211 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:18.211 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:18.212 WARNING:rgw_multi.tests:bucket lbjidp-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.13.5 target=00000000001.11.5 2020-07-17T12:07:23.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:23.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:23.215 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:23.259 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:23.259 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:23.460 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:23.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.461 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:23.461 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.461 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.462 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.462 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.462 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.462 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.463 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:23.466 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.466 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:23.466 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.466 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:07:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:07:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.11.5", 2020-07-17T12:07:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:59.064046Z" 2020-07-17T12:07:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.469 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.469 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:23.469 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.474 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.474 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.475 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.475 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.475 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.493 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.493 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.493 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.493 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.494 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.494 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.494 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.495 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.495 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:23.496 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.496 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.496 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.496 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.496 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.497 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.497 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:23.501 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:23.501 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:23.501 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:23.501 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:23.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:23.503 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.503 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.503 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:23.503 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:23.504 WARNING:rgw_multi.tests:bucket lbjidp-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.13.5 target=00000000001.11.5 2020-07-17T12:07:28.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:28.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:28.508 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:28.553 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:28.554 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:28.764 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:28.771 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.771 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:28.771 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.772 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.772 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:07:28.772 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:07:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.13.5", 2020-07-17T12:07:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.856094Z" 2020-07-17T12:07:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.779 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.780 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.780 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.780 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.781 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.784 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:28.784 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.784 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.785 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.785 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.785 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.786 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.786 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.787 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.787 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.788 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.788 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.790 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:28.790 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.790 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.791 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.791 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.791 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.792 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.792 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.793 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.793 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.794 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.794 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.794 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.795 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.795 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.796 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:28.796 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.796 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.796 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.802 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.802 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.802 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.802 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.803 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.803 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:28.803 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.803 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.803 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.804 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.804 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.804 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.804 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.805 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.805 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.805 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.807 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:28.807 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.807 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.807 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:28.811 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.811 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.811 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.814 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.814 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.814 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.815 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.816 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.816 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.816 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.817 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.817 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.817 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:28.818 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:28.818 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:28.818 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:28.819 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:28.819 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:28.819 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:28.819 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:28.819 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:28.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.820 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:28.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:28.820 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:28.821 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:28.821 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:28.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:28.821 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:28.822 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:28.823 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-13 zones={test-zone2, test-zone1} 2020-07-17T12:07:28.836 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:28.886 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:28.949 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-13 zones={test-zone2, test-zone1} 2020-07-17T12:07:28.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:28.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:28.950 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:28.986 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:28.986 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:29.181 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.13.5" 2020-07-17T12:07:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.184 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:29.185 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.187 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:29.188 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:29.189 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:29.190 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.191 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:29.191 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:29.192 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:29.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:29.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:29.195 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:29.278 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:29.278 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:29.464 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:29.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.464 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:29.465 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.465 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.465 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.465 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.465 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.466 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.466 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.466 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.466 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.467 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.467 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.468 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:29.468 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.468 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:29.468 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.469 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.469 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:07:29.469 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.469 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.469 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:29.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.470 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.470 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.13.5", 2020-07-17T12:07:29.470 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.856094Z" 2020-07-17T12:07:29.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.471 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:29.471 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.472 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.472 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.472 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.472 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.473 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.473 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.473 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.474 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.474 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.474 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.475 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:29.475 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.475 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.475 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.476 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.476 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.476 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.476 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.477 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.477 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.477 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.477 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.478 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.478 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.478 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:29.478 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.479 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.479 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.479 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.479 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.479 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.480 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.480 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.480 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.480 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.481 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.482 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:29.482 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.482 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.482 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.482 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.483 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.483 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.483 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.483 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.484 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.484 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.484 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.485 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:29.485 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.485 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.488 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.489 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.489 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.489 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.489 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.490 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.494 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.494 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.494 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.496 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:29.496 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.497 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.497 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.497 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.498 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.498 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.498 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.498 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.499 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.499 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.499 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.500 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:29.500 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.501 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.501 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.501 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.501 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.501 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.502 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.502 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.502 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.503 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.503 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.504 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:29.504 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.504 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.504 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.504 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.505 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.505 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.505 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.506 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.507 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.507 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.507 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.508 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:29.508 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:29.508 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:29.509 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:29.509 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:29.509 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:29.509 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:29.509 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:29.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.510 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:29.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.510 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:29.510 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:29.510 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:29.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.511 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:29.512 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:29.513 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-13 zones={test-zone3, test-zone1} 2020-07-17T12:07:29.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:29.578 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:29.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-13 zones={test-zone3, test-zone1} 2020-07-17T12:07:29.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:29.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:29.644 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:29.682 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:29.682 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:29.884 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.13.5" 2020-07-17T12:07:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:29.894 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:29.895 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:29.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:29.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:29.897 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:29.944 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:29.945 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:30.141 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:30.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.142 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:30.142 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.143 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.143 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.144 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.144 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.145 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.145 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.146 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.147 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.147 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.147 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.149 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:30.150 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.150 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:30.150 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.151 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.151 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.151 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.151 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.152 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.152 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.152 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.13.5", 2020-07-17T12:07:30.152 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.856094Z" 2020-07-17T12:07:30.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.153 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.153 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:30.153 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.154 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.154 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.154 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.154 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.155 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.155 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.155 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.156 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.156 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.156 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.157 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.157 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:30.157 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.158 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.158 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.158 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.159 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.159 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.159 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.160 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.160 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.160 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.160 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.161 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:30.161 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.162 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.162 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.162 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.162 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.162 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.163 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.163 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.164 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.164 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.164 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.164 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.164 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.165 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:30.165 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.165 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.166 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.166 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.166 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.166 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.166 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.167 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.167 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.167 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.167 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.168 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:30.169 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.169 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.169 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.169 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.169 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.169 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.170 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.170 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.170 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.171 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.171 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.172 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:30.172 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.172 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.172 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.172 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.173 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.173 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.173 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.173 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.174 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.174 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.174 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.174 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.174 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.175 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.176 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.176 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.176 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.176 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.176 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.177 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.177 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.177 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.177 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.178 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.178 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.178 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:30.178 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.179 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.179 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.179 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.179 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.179 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.179 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.180 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.215 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.215 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.215 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.216 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.216 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.216 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:30.217 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:30.217 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:30.217 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:30.217 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:30.217 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:30.218 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:30.218 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:30.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.218 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:30.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.219 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:30.219 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:30.219 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.220 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:30.222 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-13 2020-07-17T12:07:30.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:30.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:30.246 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:30.285 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:30.285 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:30.496 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:07:30.497 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:07:30.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.498 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:30.498 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.499 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:30.499 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000002.14.5" 2020-07-17T12:07:30.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.500 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:30.501 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.501 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:30.501 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.502 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:30.502 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.503 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.503 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:30.503 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.504 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:30.505 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.506 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:30.506 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.507 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:30.508 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.508 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.508 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:30.509 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:30.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:30.510 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:30.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:30.510 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:30.511 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:07:30.511 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:07:30.512 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-14 2020-07-17T12:07:30.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:30.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:30.518 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:30.563 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:30.564 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:30.752 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:30.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.753 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:30.753 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.753 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.753 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.753 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.754 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.754 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.754 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.754 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.755 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.755 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.756 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.757 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:30.757 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.757 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:30.757 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.757 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.757 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:07:30.758 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.758 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.758 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:30.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.759 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.759 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.14.5", 2020-07-17T12:07:30.759 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.903456Z" 2020-07-17T12:07:30.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.760 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:30.760 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.760 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.760 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.761 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.761 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.761 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.761 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.762 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.762 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.763 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.763 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.764 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:30.764 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.764 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.765 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.765 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.765 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.765 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.766 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.766 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.766 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.767 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.767 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.767 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.767 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.768 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:30.768 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.769 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.770 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.770 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.770 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.770 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.770 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.771 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.771 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.771 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.772 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.772 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:30.773 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.773 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.773 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.773 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.773 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.774 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.774 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.774 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.775 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.775 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.775 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.776 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:30.776 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.777 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.777 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.777 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.777 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.777 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.778 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.778 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.778 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.778 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.779 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.780 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:30.781 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.781 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.781 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.781 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.782 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.782 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.782 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.782 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.783 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.783 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.783 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.784 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:30.784 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.785 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.785 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.785 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.785 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.785 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.785 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.786 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.786 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.786 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.787 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.788 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:30.788 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.790 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.790 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.790 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.790 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.791 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.791 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.792 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.793 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.793 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.793 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:30.794 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:30.794 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:30.794 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:30.795 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:30.795 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:30.795 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:30.795 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:30.795 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:30.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.796 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:30.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:30.796 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:30.796 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:30.797 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:30.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:30.797 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:30.798 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-14 2020-07-17T12:07:30.798 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-14 zones={test-zone1, test-zone2} 2020-07-17T12:07:30.810 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:30.858 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:30.921 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-14 zones={test-zone1, test-zone2} 2020-07-17T12:07:30.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:30.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:30.923 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:30.961 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:30.962 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:31.161 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:07:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:07:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.161 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:31.162 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000002.14.5" 2020-07-17T12:07:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.163 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:31.165 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.166 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:31.169 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.170 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.172 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:31.176 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:07:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:31.177 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:07:31.177 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:07:31.179 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-14 2020-07-17T12:07:31.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:31.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:31.181 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:31.227 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:31.227 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:31.410 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:31.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.411 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:31.411 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.412 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.412 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.412 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.413 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.413 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.413 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.414 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.415 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.415 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.415 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.417 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:31.417 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.417 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:31.418 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.418 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.418 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:07:31.419 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.419 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.419 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:31.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.420 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.420 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.14.5", 2020-07-17T12:07:31.421 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.903456Z" 2020-07-17T12:07:31.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.424 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:31.425 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.425 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.425 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.426 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.426 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.426 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.427 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.427 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.428 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.428 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.429 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.431 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:31.431 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.431 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.432 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.432 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.432 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.433 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.433 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.434 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.434 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.435 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.435 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.437 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:31.437 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.437 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.438 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.439 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.439 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.439 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.440 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.440 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.441 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.441 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.442 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.443 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.444 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:31.444 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.445 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.445 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.445 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.446 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.446 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.446 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.447 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.448 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.448 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.448 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.450 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:31.450 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.451 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.451 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.451 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.451 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.452 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.452 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.453 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.454 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.454 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.455 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.455 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.455 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.456 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:31.457 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.457 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.457 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.457 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.457 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.458 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.458 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.458 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.459 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.459 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.459 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.461 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:31.462 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.462 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.464 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.464 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.464 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.465 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.465 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.465 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.466 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.466 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.466 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.467 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:31.467 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.467 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.467 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.468 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.468 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.468 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.468 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.470 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.470 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.470 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.471 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.471 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:31.472 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:31.472 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:31.472 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:31.472 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:31.472 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:31.473 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:31.473 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:31.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.473 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:31.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.474 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:31.475 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:31.475 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:31.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.476 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:31.476 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-14 2020-07-17T12:07:31.477 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-14 zones={test-zone3, test-zone2} 2020-07-17T12:07:31.489 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:31.538 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:31.601 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-14 zones={test-zone3, test-zone2} 2020-07-17T12:07:31.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:31.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:31.628 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:31.666 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:31.666 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:31.861 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.11.5" 2020-07-17T12:07:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:31.870 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:31.870 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-15 2020-07-17T12:07:31.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:31.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:31.872 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:31.956 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:31.957 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:32.138 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:32.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.139 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.142 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "myobj", 2020-07-17T12:07:32.143 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.144 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.11.5", 2020-07-17T12:07:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.959153Z" 2020-07-17T12:07:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.146 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.148 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.151 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:32.152 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.153 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.154 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.156 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.157 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.158 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.160 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.162 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:32.163 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.164 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.165 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.168 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.170 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.173 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:32.178 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:32.179 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:32.180 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.181 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:32.182 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-15 2020-07-17T12:07:32.182 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-15 zones={test-zone1, test-zone3} 2020-07-17T12:07:32.194 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:32.242 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:32.309 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-15 zones={test-zone1, test-zone3} 2020-07-17T12:07:32.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:32.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:32.310 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:32.350 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:32.350 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:32.554 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:32.554 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:32.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.555 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:32.555 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.556 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:32.556 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.11.5" 2020-07-17T12:07:32.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.556 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:32.557 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.557 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:32.557 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.558 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:32.558 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.559 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:32.559 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.559 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:32.560 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.560 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:32.560 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.561 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:32.561 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.562 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:32.562 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:32.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:32.562 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:32.562 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:32.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:32.563 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:32.563 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:32.564 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-15 2020-07-17T12:07:32.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:32.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:32.566 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:32.615 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:32.615 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:32.816 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:33.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.023 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:33.024 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.024 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.024 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.024 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.024 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.025 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.025 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.025 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.025 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.026 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.026 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.027 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:33.027 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.027 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:33.027 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.027 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.028 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "myobj", 2020-07-17T12:07:33.028 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.028 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.028 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:07:33.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.029 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.029 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.11.5", 2020-07-17T12:07:33.029 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.959153Z" 2020-07-17T12:07:33.029 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.030 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.030 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:33.030 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.030 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.031 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.031 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.031 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.031 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.031 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.032 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.032 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.032 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.032 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.033 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:33.034 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.034 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.034 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.034 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.034 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.034 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.035 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.035 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.036 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.036 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.037 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:33.037 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.037 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.037 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.037 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.038 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.038 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.038 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.039 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.039 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.039 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.040 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:33.040 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.040 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.040 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.041 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.041 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.041 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.041 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.043 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.043 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.043 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.057 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.058 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:33.058 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.059 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.059 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.059 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.059 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.059 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.060 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.060 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.060 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.061 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.061 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.062 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:33.062 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.062 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.062 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.062 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.063 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.063 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.063 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.063 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.064 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.064 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.064 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.065 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:33.065 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.065 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.065 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.066 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.066 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.066 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.066 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.066 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.067 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.067 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.067 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.068 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:33.068 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.068 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.069 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.069 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.069 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.069 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.069 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.074 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.078 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:33.079 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.079 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:33.079 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:33.080 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:33.080 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:33.080 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:33.081 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:33.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.081 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:33.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.082 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:33.082 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:33.083 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:33.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.083 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:33.084 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-15 2020-07-17T12:07:33.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-15 zones={test-zone2, test-zone3} 2020-07-17T12:07:33.098 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:07:33.146 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:07:33.213 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-15 zones={test-zone2, test-zone3} 2020-07-17T12:07:33.214 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-07-17T12:07:33.215 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-16 2020-07-17T12:07:33.245 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-17 2020-07-17T12:07:33.300 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-18 2020-07-17T12:07:33.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:33.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:33.391 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:33.429 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:33.429 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:33.586 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:07:33.617 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987653.381436_89.1", 2020-07-17T12:07:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:07:33.381436Z" 2020-07-17T12:07:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:33.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:07:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:07:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:33.619 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:07:33.620 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:07:33.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:33.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:33.621 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:07:33.621 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:07:33.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:33.622 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:33.623 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:07:33.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:33.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:33.625 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:33.671 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:33.672 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:33.884 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:07:33.884 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:07:33.884 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:07:33.884 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:07:33.885 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:07:33.885 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:07:33.885 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:33.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.885 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:07:33.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.886 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:33.886 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.887 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:33.887 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987653.325975_81.1", 2020-07-17T12:07:33.887 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:33.887 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:07:33.888 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:33.888 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.325975Z", 2020-07-17T12:07:33.888 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:33.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.889 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:33.889 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.889 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:33.889 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:07:33.890 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:33.890 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:33.890 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:33.890 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:07:33.891 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:33.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.891 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:33.891 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.892 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:33.892 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:07:33.892 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:33.892 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:33.892 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:33.893 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:07:33.893 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:33.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:33.894 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:33.894 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:33.894 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:33.894 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:07:33.895 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:33.895 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:33.895 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:33.895 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:07:33.895 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:33.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.896 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:07:33.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:33.897 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:07:33.897 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:07:33.897 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:07:33.897 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:33.898 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:07:33.899 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594987653.381436_89.1 target=1_1594987653.325975_81.1 2020-07-17T12:07:38.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:38.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:38.902 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:38.939 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:38.940 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:39.154 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:07:39.155 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:07:39.155 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:07:39.155 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:07:39.155 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:07:39.156 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:07:39.156 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.156 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:07:39.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:39.157 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:39.157 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:39.157 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:39.157 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987653.381436_89.1", 2020-07-17T12:07:39.157 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:39.158 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:07:39.158 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:39.158 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.381436Z", 2020-07-17T12:07:39.158 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.158 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.159 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:39.159 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:39.159 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:39.159 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:39.159 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:07:39.160 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:39.160 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:39.160 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:39.160 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:07:39.160 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:39.161 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:39.161 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:39.162 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:39.162 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:07:39.162 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:39.162 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:39.162 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:39.162 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:07:39.163 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.163 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:39.163 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:39.164 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:39.164 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:39.164 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:07:39.164 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:39.164 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:39.165 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:39.165 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:07:39.165 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.165 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.165 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.167 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:07:39.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.168 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:07:39.168 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:07:39.168 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:07:39.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.168 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:07:39.169 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:07:39.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:07:39.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:39.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:39.172 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:39.215 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:39.216 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:39.394 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:39.395 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:07:39.395 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:07:39.395 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:07:39.396 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:07:39.396 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:07:39.396 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:07:39.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:39.397 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:39.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:39.398 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:39.398 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:39.399 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:07:39.399 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987653.381436_89.1", 2020-07-17T12:07:39.399 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:07:39.400 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:07:39.400 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:07:39.400 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.381436Z", 2020-07-17T12:07:39.401 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:07:39.401 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:39.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:39.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:39.402 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:39.402 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:39.403 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:07:39.403 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:07:39.403 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:07:39.404 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:07:39.404 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:07:39.404 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:07:39.405 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:07:39.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:39.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:39.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:39.407 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:39.407 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:39.407 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:07:39.408 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:07:39.408 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:07:39.408 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:07:39.409 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:07:39.409 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:07:39.410 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:07:39.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:39.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:39.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:39.411 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:39.412 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:39.412 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:07:39.413 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:07:39.413 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:07:39.413 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:07:39.414 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:07:39.414 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:07:39.414 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:07:39.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:39.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:39.415 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:39.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:39.416 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:07:39.417 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:07:39.417 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:07:39.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:39.418 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:39.419 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:07:39.420 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:07:39.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:39.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:39.423 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:39.467 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:39.467 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:39.745 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:07:39.746 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:07:39.746 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:07:39.746 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:07:39.746 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:07:39.747 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:07:39.747 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.748 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:07:39.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:39.748 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:39.748 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:39.748 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:39.749 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987653.381436_89.1", 2020-07-17T12:07:39.749 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:39.749 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:07:39.750 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:39.750 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.381436Z", 2020-07-17T12:07:39.750 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.750 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:39.751 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:39.752 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:39.752 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:39.752 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:07:39.752 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:39.753 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:39.753 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:39.753 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:07:39.754 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.754 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:39.755 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:39.755 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:39.756 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:39.756 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:07:39.756 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:39.756 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:39.757 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:39.757 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:07:39.757 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:39.761 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:39.761 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:39.761 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:07:39.761 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:07:39.762 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:07:39.762 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:07:39.762 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:07:39.763 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:07:39.763 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:07:39.763 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.764 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.764 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:07:39.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:39.765 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:07:39.765 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:07:39.765 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:07:39.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:39.766 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:07:39.767 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:07:39.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:39.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:39.820 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:39.861 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:39.861 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:40.050 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.9.10" 2020-07-17T12:07:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:40.064 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:40.065 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:07:40.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:40.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:40.067 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:40.119 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:40.119 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:40.295 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:40.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.296 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:40.296 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.296 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.296 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.296 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.297 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.297 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.297 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.297 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.298 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.298 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.298 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.298 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.299 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.299 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:40.299 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.299 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.300 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.300 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.300 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.300 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.300 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.301 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.301 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.301 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.302 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.302 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.303 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.303 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:40.303 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.303 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.304 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.304 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.304 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.304 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.304 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.305 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.305 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.305 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.306 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.306 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:40.307 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.307 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.307 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.307 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.307 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.309 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.309 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.310 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:40.310 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.310 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.310 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.310 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.312 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.312 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.312 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.312 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.312 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.313 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.313 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:40.313 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.313 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:40.314 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.314 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.314 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:07:40.314 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:07:40.314 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.315 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:07:40.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.315 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.315 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.9.10", 2020-07-17T12:07:40.315 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:39.883965Z" 2020-07-17T12:07:40.316 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.316 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.316 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:40.316 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.317 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.317 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.317 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.318 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.318 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.318 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.318 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.319 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.319 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.319 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.320 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:40.320 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.320 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.320 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.321 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.321 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.321 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.321 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.322 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.322 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.322 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.322 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.323 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:40.323 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.323 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.324 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.324 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.324 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.324 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.324 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.325 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.325 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.325 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.326 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.327 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:40.327 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.327 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.327 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.327 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.328 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.328 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.328 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.329 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.330 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.330 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.330 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:40.331 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:40.331 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:40.331 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:40.332 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:40.332 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:40.332 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:40.332 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:40.332 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:40.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.333 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:40.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:40.333 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:40.333 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:40.333 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:40.334 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:40.335 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:07:40.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:40.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:40.336 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:40.374 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:40.374 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:40.568 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:40.569 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:40.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.569 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:40.570 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.571 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:40.571 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.572 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:40.572 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.573 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:40.574 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.574 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:40.574 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.575 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:40.575 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000005.15.9" 2020-07-17T12:07:40.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.576 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:40.576 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.576 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:40.577 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.577 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:40.577 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.578 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:40.578 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.579 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:40.579 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:40.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.579 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:40.579 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:40.580 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:07:40.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:40.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:40.582 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:40.664 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:40.665 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:40.865 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:40.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.865 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:40.866 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.866 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.866 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.866 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.866 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.867 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.867 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.867 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.868 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.868 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.868 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.869 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:40.869 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.869 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.870 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.870 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.870 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.870 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.870 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.871 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.871 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.871 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.872 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.873 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:40.874 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.874 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.874 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.874 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.874 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.875 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.875 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.875 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.876 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.876 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.876 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.877 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:40.877 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.878 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.878 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.878 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.878 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.878 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.887 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.887 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.889 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.889 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.889 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.890 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:40.891 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.891 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.891 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.892 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.892 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.893 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.893 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.893 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.894 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.894 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.894 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.895 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:40.896 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.896 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:40.896 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.896 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.896 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:07:40.897 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:07:40.897 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.897 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:40.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.898 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.898 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.9.10", 2020-07-17T12:07:40.898 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:39.886029Z" 2020-07-17T12:07:40.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.899 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:40.899 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.900 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.900 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.900 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.900 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.900 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.901 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.901 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.901 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.902 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.902 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.902 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.902 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.903 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:40.903 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.903 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.903 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.904 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.904 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.904 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.904 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.905 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.905 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.905 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.906 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.907 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:40.907 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.910 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.911 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.911 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.911 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.911 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.911 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.912 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.912 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.912 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.913 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.914 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:40.914 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.914 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.914 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.914 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:40.922 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:40.923 WARNING:rgw_multi.tests:bucket lbjidp-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.15.9 target=00000000002.9.10 2020-07-17T12:07:45.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:45.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:45.926 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:45.970 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:45.971 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:46.164 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:07:46.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.165 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:46.165 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.165 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.165 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.166 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.166 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.166 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.166 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:46.170 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.171 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.171 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.171 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.171 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.171 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.172 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.173 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.173 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.174 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.174 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.175 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:46.175 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.175 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.175 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.176 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.176 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.176 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.176 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.177 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.177 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.177 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.178 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.179 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:46.179 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.179 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.179 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.180 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.180 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.180 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.180 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.180 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.181 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.181 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.181 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.182 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:46.183 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.183 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.183 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.183 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.183 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.184 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.184 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.184 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.184 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.185 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.185 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.186 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:46.186 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.186 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:07:46.187 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.187 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.187 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:07:46.187 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:07:46.188 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.188 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:07:46.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.189 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.189 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000005.15.9", 2020-07-17T12:07:46.189 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:39.776659Z" 2020-07-17T12:07:46.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.190 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:46.190 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.190 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.190 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.191 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.191 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.191 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.191 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.192 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.192 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.192 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.192 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.193 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.193 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.193 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:46.194 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.194 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.194 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.194 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.195 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.195 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.195 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.195 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.196 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.196 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.197 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.198 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:46.198 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.198 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.198 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.198 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.199 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.199 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.199 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.200 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.200 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.200 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.201 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.201 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.201 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.202 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:46.202 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.203 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.203 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.203 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.203 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.203 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.203 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.205 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.205 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.206 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.206 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.207 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:46.207 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:07:46.207 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:07:46.207 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:07:46.208 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:07:46.208 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:07:46.208 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:07:46.208 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:07:46.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.209 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:07:46.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.209 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:07:46.209 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:07:46.209 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.210 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:07:46.211 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:07:46.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:46.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:46.213 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:46.290 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:46.291 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:46.483 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:07:46.483 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:07:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:07:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.485 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:07:46.485 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.485 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:07:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:07:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.487 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:07:46.487 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.488 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:07:46.488 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000006.17.9" 2020-07-17T12:07:46.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.489 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:07:46.489 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.490 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:07:46.490 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.490 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:07:46.491 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.491 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:07:46.491 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:07:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:07:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:07:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:07:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:07:46.493 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:07:46.493 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:07:46.494 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:07:46.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:46.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:46.496 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:46.542 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:46.543 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:46.754 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:46.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.755 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:46.755 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.756 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.756 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.756 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.756 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.756 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.757 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.757 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.760 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.760 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.760 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.760 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.760 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.761 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.761 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.762 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.762 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.762 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.762 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.763 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:46.763 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.763 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.763 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.764 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.764 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.764 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.764 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.764 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.765 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.765 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.765 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.765 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.766 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:46.767 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.767 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.767 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.767 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.767 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.768 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.768 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.768 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.769 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.769 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.769 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.770 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:46.770 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.771 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.771 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.771 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.771 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.771 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.772 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.772 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.773 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.773 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.773 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.775 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:46.775 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.775 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:46.775 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.776 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.776 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.776 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.777 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.777 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.777 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.778 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000005.15.9", 2020-07-17T12:07:46.778 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:39.776659Z" 2020-07-17T12:07:46.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.779 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:46.779 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.779 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.779 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.780 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.780 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.780 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.780 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.780 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.781 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.781 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.781 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.782 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.783 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:46.783 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.783 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.783 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.784 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.784 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.784 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.784 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.785 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.785 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.785 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.786 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.787 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:46.787 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.787 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.787 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.787 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.787 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.788 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.788 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.788 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.789 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.789 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.789 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.790 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:46.790 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.790 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.790 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.791 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.791 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.791 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.791 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.792 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.815 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.815 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.815 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.815 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.815 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:46.816 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:46.816 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:46.817 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:46.817 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:46.817 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:46.817 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:46.818 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:46.818 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:46.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.818 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:46.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:46.819 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:46.819 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:46.819 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:46.819 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:46.820 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:46.823 WARNING:rgw_multi.tests:bucket lbjidp-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.17.9 target=00000000005.15.9 2020-07-17T12:07:51.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:51.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:51.826 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:51.870 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:51.870 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:52.083 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000005.15.9", 2020-07-17T12:07:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:39.776659Z" 2020-07-17T12:07:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:52.154 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:52.156 WARNING:rgw_multi.tests:bucket lbjidp-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.17.9 target=00000000005.15.9 2020-07-17T12:07:57.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:57.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:07:57.178 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:07:57.229 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:07:57.230 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:07:57.433 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:07:57.433 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.433 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:07:57.433 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.434 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.434 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.434 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.434 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.436 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:07:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.437 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.438 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.439 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.439 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.439 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.439 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.440 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:07:57.441 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.441 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.441 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.441 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.441 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.442 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.442 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.442 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.443 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.443 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.443 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:07:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.444 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.445 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.448 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.448 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.449 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.449 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.449 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.449 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.450 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.450 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:07:57.450 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.450 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.450 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.451 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.451 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.451 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.451 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.452 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.452 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.452 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.452 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.453 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.453 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.453 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:07:57.454 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.454 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:07:57.454 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.454 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.454 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.454 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.455 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.455 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.455 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.456 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000005.15.9", 2020-07-17T12:07:57.456 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:39.776659Z" 2020-07-17T12:07:57.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.457 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:07:57.457 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.457 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.457 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.458 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.458 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.458 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.458 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.459 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.459 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.459 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.459 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.460 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:07:57.460 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.461 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.461 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.461 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.461 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.461 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.462 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.462 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.462 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.463 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.463 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.463 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.463 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.464 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:07:57.464 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.464 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.464 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.465 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.465 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.465 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.465 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.477 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.478 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.478 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.478 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.478 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.479 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.479 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:07:57.479 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.480 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.480 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.480 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.480 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.480 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.481 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.481 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.501 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.501 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.501 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.501 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.501 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:07:57.502 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:07:57.502 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:07:57.502 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:07:57.503 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:07:57.503 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:07:57.503 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:07:57.503 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:07:57.503 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:07:57.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.504 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:07:57.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:07:57.504 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:07:57.504 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:07:57.505 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:07:57.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:07:57.505 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:07:57.506 WARNING:rgw_multi.tests:bucket lbjidp-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.17.9 target=00000000005.15.9 2020-07-17T12:08:02.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:02.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:02.509 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:02.555 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:02.556 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:02.762 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:08:02.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.762 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:02.763 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.763 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.763 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.764 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.764 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.764 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.765 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.765 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.766 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.766 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.766 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.768 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:02.768 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.769 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.769 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.769 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.770 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.770 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.770 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.771 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.772 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.772 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.772 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.774 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:02.774 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.775 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.775 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.775 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.776 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.776 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.776 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.777 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.778 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.778 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.779 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.780 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:02.780 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.780 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.781 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.781 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.781 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.781 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.781 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.782 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.782 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.783 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.783 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.783 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.783 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.784 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:02.784 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.784 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.784 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.784 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.785 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.785 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.785 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.786 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.786 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.786 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.786 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.787 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.787 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.787 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:02.787 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.788 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:02.788 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.788 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.788 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.788 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.788 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.789 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.789 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.789 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000005.15.9", 2020-07-17T12:08:02.790 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:39.776659Z" 2020-07-17T12:08:02.790 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.790 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.790 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:02.791 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.791 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.791 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.791 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.791 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.791 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.792 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.792 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.792 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.793 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.793 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.793 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.793 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.794 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:02.794 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.794 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.794 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.794 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.795 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.795 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.795 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.795 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.796 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.796 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.796 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.796 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.797 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.797 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:02.797 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.798 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.798 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.798 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.798 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.798 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.798 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.799 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.799 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.799 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.799 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.800 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.800 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.800 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.802 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.802 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.830 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.830 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.830 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:02.831 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:02.831 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:02.832 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:02.832 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:02.832 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:02.832 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:02.832 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:02.833 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:02.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.833 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:02.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:02.833 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:02.833 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:02.834 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:02.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:02.834 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:08:02.835 WARNING:rgw_multi.tests:bucket lbjidp-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.17.9 target=00000000005.15.9 2020-07-17T12:08:07.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:07.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:07.838 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:07.885 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:07.886 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:08.083 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:08:08.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.084 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:08.084 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.084 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.084 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.085 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.085 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.085 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.085 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.086 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.086 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.086 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.086 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.087 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:08.087 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.088 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.088 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.088 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.088 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.088 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.088 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.089 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.089 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.089 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.090 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.090 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:08.091 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.091 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.091 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.091 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.091 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.092 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.092 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.092 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.092 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.093 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.093 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.094 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:08.094 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.094 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.094 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.095 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.095 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.095 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.095 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.096 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.096 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.096 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.097 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.098 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:08.098 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.098 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.098 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.098 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.109 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.109 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.109 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.110 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.110 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.110 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.111 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:08.112 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.112 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:08.112 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.112 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.112 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.113 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.113 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.113 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.114 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.114 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.17.9", 2020-07-17T12:08:08.114 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:39.776659Z" 2020-07-17T12:08:08.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.115 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:08.115 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.115 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.115 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.116 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.116 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.116 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.116 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.116 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.117 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.117 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.117 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.117 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.118 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:08.118 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.118 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.119 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.119 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.119 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.119 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.119 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.120 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.120 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.121 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.121 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.122 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:08.122 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.122 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.122 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.122 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.123 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.123 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.123 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.123 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.124 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.124 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.124 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.124 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.124 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.125 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:08.125 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.125 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.125 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.126 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.126 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.126 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.126 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.126 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.149 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.150 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.150 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.151 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:08.151 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:08.151 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:08.151 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:08.151 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.152 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:08.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.153 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:08.153 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:08.153 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.153 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.153 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.154 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.154 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:08:08.155 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:08:08.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:08.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:08.207 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:08.248 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:08.249 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:08.453 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:08:08.453 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:08:08.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.454 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:08.454 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.454 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:08.455 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.455 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:08.455 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.456 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:08.456 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.457 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:08.457 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.457 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:08.457 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000002.11.10" 2020-07-17T12:08:08.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.458 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:08.458 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.459 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:08.459 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.460 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:08.460 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.460 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:08.460 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.461 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:08.461 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.462 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:08:08.462 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:08:08.463 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:08:08.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:08.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:08.464 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:08.516 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:08.517 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:08.687 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:08:08.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.687 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:08.688 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.688 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.688 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.688 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.688 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.689 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.689 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.689 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.689 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.690 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.690 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.691 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:08.691 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.691 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.691 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.692 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.692 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.692 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.692 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.693 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.693 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.693 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.694 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.694 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:08.694 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.694 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.695 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.695 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.695 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.695 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.696 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.696 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.696 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.696 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.697 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:08.698 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.698 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.698 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.698 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.698 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.698 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.699 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.699 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.699 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.700 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.700 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.701 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:08.701 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.701 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.701 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.701 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.702 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.702 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.702 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.702 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.703 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.703 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.703 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.704 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:08.704 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.704 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:08:08.705 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.705 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.705 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:08:08.705 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:08:08.705 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.706 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:08:08.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.706 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.706 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.11.10", 2020-07-17T12:08:08.706 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:08.365991Z" 2020-07-17T12:08:08.707 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.707 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.707 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:08.707 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.708 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.708 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.708 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.708 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.708 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.709 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.709 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.709 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.709 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.710 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.711 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:08.711 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.711 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.711 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.711 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.712 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.712 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.712 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.712 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.713 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.713 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.713 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.714 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:08.714 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.714 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.714 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.715 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.715 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.715 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.715 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.715 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.716 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.716 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.716 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.717 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:08.717 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.717 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.718 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.718 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.718 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.718 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.718 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.720 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.720 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.720 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.720 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:08.721 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:08.721 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:08.722 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:08.722 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:08.722 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:08.722 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:08.722 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:08.722 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:08.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.723 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:08.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:08.723 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:08.723 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:08.724 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:08.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:08.724 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:08:08.725 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:08:08.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:08.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:08.726 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:08.769 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:08.769 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:08.968 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:08:08.968 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:08:08.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.969 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:08.969 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.970 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:08.970 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.971 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:08.971 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.971 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:08.972 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.972 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:08.972 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.973 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:08.973 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000005.17.9" 2020-07-17T12:08:08.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.974 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:08.974 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.974 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:08.975 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.975 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:08.975 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.976 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:08.976 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:08.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:08.977 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:08.977 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:08:08.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:08.977 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:08:08.977 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:08:08.979 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:08:08.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:08.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:08.980 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:09.025 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:09.199 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:08:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.199 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.200 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.201 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.203 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.204 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.205 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.207 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.210 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.211 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.211 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.215 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.215 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.215 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.216 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.216 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.216 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.217 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.217 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:09.218 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.218 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.218 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.218 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.218 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.219 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.219 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.219 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.220 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.220 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.220 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.220 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.220 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.221 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:09.221 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.221 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:08:09.221 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.222 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.222 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:08:09.222 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:08:09.222 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.222 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:08:09.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.223 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.223 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.11.10", 2020-07-17T12:08:09.223 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:08.370013Z" 2020-07-17T12:08:09.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.224 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:09.224 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.224 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.225 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.225 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.225 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.225 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.225 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.230 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.230 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.230 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.230 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.231 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.231 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.231 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:09.232 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.232 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.232 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.232 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.232 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.232 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.233 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.233 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.233 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.234 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.234 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.234 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.234 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.235 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:09.235 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.235 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.235 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.235 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.235 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.236 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.240 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.241 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.242 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.242 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.242 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.243 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.243 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.244 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:09.244 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.245 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.245 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.245 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.246 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.246 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.246 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.252 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.253 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.253 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.253 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:09.254 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:09.255 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:09.255 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:09.255 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:09.255 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:09.255 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:09.255 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:09.256 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:09.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.256 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:09.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:09.256 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:09.257 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:09.257 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:09.258 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.258 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.258 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:09.258 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:08:09.259 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000005.17.9 target=00000000002.11.10 2020-07-17T12:08:14.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:14.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:14.263 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:14.309 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:14.309 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:14.490 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:08:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.501 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.504 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:14.507 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:08:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:08:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:08:14.508 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:08:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.11.10", 2020-07-17T12:08:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:08.370013Z" 2020-07-17T12:08:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:14.511 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.511 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.511 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.511 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.518 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:14.529 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:08:14.529 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000005.17.9 target=00000000002.11.10 2020-07-17T12:08:19.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:19.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:19.532 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:19.581 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:19.582 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:19.778 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:08:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.779 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.781 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.782 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.786 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.791 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:19.792 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.794 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.795 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:08:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:08:19.796 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:08:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:08:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.11.10", 2020-07-17T12:08:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:08.370013Z" 2020-07-17T12:08:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.802 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.804 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.806 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.809 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.810 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:19.816 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:08:19.816 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000005.17.9 target=00000000002.11.10 2020-07-17T12:08:24.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:24.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:24.822 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:24.867 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:25.062 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:08:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.078 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:25.078 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.078 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.078 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.078 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.079 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.079 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.079 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.079 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.080 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.080 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.080 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.081 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:25.082 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.082 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:08:25.082 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.082 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.082 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:08:25.083 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:08:25.083 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.084 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:08:25.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.084 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.084 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.11.10", 2020-07-17T12:08:25.084 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:08.370013Z" 2020-07-17T12:08:25.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.085 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:25.086 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.086 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.086 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.086 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.086 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.087 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.087 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.087 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.088 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.088 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.089 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.090 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:25.090 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.090 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.091 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.091 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.091 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.091 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.091 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.092 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.092 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.092 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.093 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.094 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:25.094 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.094 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.094 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.094 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.095 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.095 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.095 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.095 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.096 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.096 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.096 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.097 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:25.097 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.097 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.098 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.098 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.098 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.098 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.098 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.100 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.101 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.101 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.101 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:25.102 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:25.103 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:25.103 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:25.103 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:25.103 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:25.103 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:25.104 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:25.104 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:25.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.104 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:25.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:25.105 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:25.105 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:25.105 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:25.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:25.106 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:08:25.107 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000005.17.9 target=00000000002.11.10 2020-07-17T12:08:30.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:30.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:30.110 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:30.158 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:30.159 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:30.338 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:08:30.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.339 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:30.339 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.339 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.340 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.340 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.340 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.340 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.340 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.341 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.341 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.341 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.341 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.342 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:30.343 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.343 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.343 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.343 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.343 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.343 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.344 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.344 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.344 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.345 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.345 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.346 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:30.346 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.346 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.347 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.347 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.347 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.348 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.348 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.349 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.349 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.349 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.350 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.351 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:30.352 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.352 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.352 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.353 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.353 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.353 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.354 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.354 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.355 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.355 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.355 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.356 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:30.357 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.357 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.357 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.358 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.358 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.358 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.358 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.359 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.359 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.359 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.359 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.361 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:30.361 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.361 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:08:30.361 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.362 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.362 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:08:30.362 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:08:30.362 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.363 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:08:30.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.363 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.364 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000006.19.9", 2020-07-17T12:08:30.364 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:08.162845Z" 2020-07-17T12:08:30.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.365 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:30.365 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.365 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.366 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.366 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.366 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.366 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.366 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.367 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.367 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.367 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.368 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.369 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:30.369 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.369 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.369 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.369 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.369 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.370 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.370 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.370 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.371 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.371 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.371 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.372 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:30.372 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.372 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.372 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.372 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.373 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.373 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.373 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.373 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.374 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.374 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.374 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.375 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:30.375 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.376 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.376 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.376 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.376 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.376 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.377 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.378 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.379 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.379 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.379 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.380 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:30.380 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.380 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.381 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.381 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.381 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.381 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.381 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.382 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.382 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.382 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.382 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.383 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:08:30.384 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:08:30.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:30.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:30.436 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:30.476 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:30.476 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:30.666 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:08:30.666 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:08:30.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.667 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:30.667 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.668 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:30.668 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.668 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:30.669 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.669 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:30.669 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.670 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:30.670 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.671 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:30.671 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.15.10" 2020-07-17T12:08:30.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.671 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:30.671 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.672 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:30.672 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.673 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:30.673 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.673 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:30.674 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.674 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:30.674 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:30.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.675 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:08:30.675 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:08:30.676 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:08:30.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:30.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:30.677 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:30.761 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:30.762 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:30.932 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:08:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:08:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:08:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:08:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:08:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.15.10", 2020-07-17T12:08:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:30.593965Z" 2020-07-17T12:08:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:08:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:08:30.971 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:08:30.971 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:08:30.971 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:08:30.971 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:08:30.971 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:08:30.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.972 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:08:30.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:30.972 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:08:30.973 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:08:30.973 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:08:30.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:30.974 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:08:30.974 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:08:30.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:30.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:30.976 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:31.054 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:31.054 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:31.248 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:08:31.248 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:08:31.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.249 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:31.249 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.250 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:31.250 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.250 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:31.251 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.251 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:31.251 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.252 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:31.252 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.253 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:31.253 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.19.9" 2020-07-17T12:08:31.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.253 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:31.254 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.254 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:31.254 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.255 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:31.255 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.256 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:31.256 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:31.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:31.256 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:31.257 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:31.257 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:31.257 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:08:31.257 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:08:31.258 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:08:31.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:31.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:31.260 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:31.306 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:31.307 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:31.518 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:08:31.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.518 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:31.518 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.519 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.519 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.519 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.519 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.519 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.520 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.520 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.520 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.520 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.521 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.522 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:31.522 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.523 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.523 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.523 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.523 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.523 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.523 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.524 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.524 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.524 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.525 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.525 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.525 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.526 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:31.526 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.526 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.526 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.526 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.526 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.527 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.527 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.527 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.528 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.528 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.528 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.529 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.530 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:31.531 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.531 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.531 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.531 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.531 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.531 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.532 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.532 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.532 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.533 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.533 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.534 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:31.539 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.539 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.539 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.539 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.540 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.540 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.540 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.540 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.541 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.541 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.541 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.542 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:31.543 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.543 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:31.544 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.544 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.544 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:08:31.544 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:08:31.544 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.545 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:08:31.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.545 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.545 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.15.10", 2020-07-17T12:08:31.545 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:30.595925Z" 2020-07-17T12:08:31.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.549 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:31.549 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.550 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.550 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.550 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.550 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.550 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.550 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.551 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.551 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.551 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.552 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.552 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.552 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.552 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:31.553 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.553 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.553 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.553 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.553 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.554 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.554 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.554 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.554 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.555 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.555 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.556 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:31.556 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.556 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.556 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.556 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.557 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.557 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.557 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.557 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.558 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.558 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.558 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.558 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.559 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.559 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:31.559 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.560 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.560 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.560 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.560 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.560 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.560 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.562 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.562 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.562 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.562 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:31.563 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:31.564 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:31.564 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:31.564 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:31.564 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:31.564 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:31.564 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:31.565 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:31.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.565 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:31.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:31.565 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:31.566 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:31.566 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:31.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:31.566 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:08:31.567 WARNING:rgw_multi.tests:bucket lbjidp-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000004.19.9 target=00000000002.15.10 2020-07-17T12:08:36.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:36.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:36.570 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:36.615 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:36.615 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:36.814 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:08:36.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.815 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:36.815 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.815 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.816 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.816 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.816 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.816 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.816 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.817 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.817 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.817 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.818 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:36.819 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.819 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.819 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.819 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.819 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.820 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.820 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.820 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.820 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.821 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.821 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.822 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:36.822 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.822 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.822 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.822 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.823 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.823 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.823 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.823 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.824 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.824 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.824 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.825 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:36.825 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.825 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.826 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.826 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.827 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.827 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.827 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.827 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.828 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.828 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.828 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.828 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.828 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.829 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:36.829 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.829 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.829 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.830 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.830 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.830 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.830 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.831 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.831 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.831 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.831 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.832 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:36.832 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.833 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:36.833 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.833 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.833 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:08:36.834 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:08:36.834 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.843 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:08:36.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.843 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.843 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.15.10", 2020-07-17T12:08:36.843 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:30.595925Z" 2020-07-17T12:08:36.844 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.844 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.845 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:36.845 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.845 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.845 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.845 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.846 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.846 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.846 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.846 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.847 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.847 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.847 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.847 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.847 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.848 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:36.848 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.848 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.849 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.849 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.849 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.849 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.849 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.850 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.850 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.850 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.850 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.851 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:36.851 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.852 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.852 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.852 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.852 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.852 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.853 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.853 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.853 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.853 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.854 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.854 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.854 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.855 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:36.855 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.855 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.855 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.855 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.855 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.856 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.856 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.857 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.857 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.858 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.858 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:36.859 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:36.859 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:36.860 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:36.860 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:36.860 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:36.860 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:36.860 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:36.860 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:36.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.861 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:36.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:36.861 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:36.861 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:36.862 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:08:36.863 WARNING:rgw_multi.tests:bucket lbjidp-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000004.19.9 target=00000000002.15.10 2020-07-17T12:08:41.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:41.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:41.866 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:41.911 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:41.912 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:42.125 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:08:42.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.125 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:42.126 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.126 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.126 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.126 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.127 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.127 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.127 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.128 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.128 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.129 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.129 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.130 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:42.131 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.131 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.131 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.131 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.131 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.131 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.132 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.132 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.133 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.133 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.133 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.134 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:42.134 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.134 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.135 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.135 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.135 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.135 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.135 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.136 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.136 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.136 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.137 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.137 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.137 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.138 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:42.138 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.138 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.138 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.139 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.139 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.139 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.139 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.140 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.140 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.140 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.140 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.141 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:42.141 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.142 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.142 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.142 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.142 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.142 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.142 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.143 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.143 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.144 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.144 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.145 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:42.145 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.145 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:42.146 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.146 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.146 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:08:42.146 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:08:42.146 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.147 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:08:42.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.147 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.147 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.15.10", 2020-07-17T12:08:42.147 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:30.595925Z" 2020-07-17T12:08:42.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.148 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:42.149 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.149 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.149 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.149 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.149 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.149 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.150 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.150 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.151 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.151 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.151 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.152 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:42.152 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.152 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.153 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.153 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.153 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.154 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.154 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.154 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.155 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.155 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.156 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.157 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.157 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:42.157 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.158 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.158 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.158 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.158 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.159 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.159 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.159 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.159 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.160 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.160 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.160 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.160 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.161 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:42.161 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.161 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.161 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.162 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.162 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.162 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.163 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.164 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.164 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.165 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.165 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.165 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.165 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:42.166 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:42.166 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:42.166 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:42.166 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:42.167 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:42.167 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:42.167 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:42.167 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:42.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.168 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:42.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:42.168 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:42.168 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:42.168 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:42.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:42.169 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:08:42.169 WARNING:rgw_multi.tests:bucket lbjidp-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000004.19.9 target=00000000002.15.10 2020-07-17T12:08:47.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:47.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:47.173 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:47.218 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:47.218 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:47.429 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:08:47.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.429 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:47.429 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.430 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.430 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.430 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.430 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.431 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.431 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.432 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.432 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.433 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.433 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.434 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.434 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:47.435 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.435 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.435 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.436 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.436 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.436 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.436 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.437 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.437 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.437 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.438 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.438 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.438 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.438 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.442 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:47.443 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.443 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.443 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.443 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.443 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.444 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.444 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.444 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.445 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.445 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.446 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.446 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.446 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.447 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:47.448 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.448 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.448 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.448 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.449 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.449 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.449 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.450 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.450 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.451 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.451 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.451 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.453 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:47.453 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.453 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.454 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.454 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.455 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.455 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.455 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.456 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.457 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.457 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.457 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.458 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.458 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.459 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:47.459 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.460 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:47.460 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.460 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.461 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:08:47.461 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:08:47.461 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.462 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:08:47.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.462 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.463 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.15.10", 2020-07-17T12:08:47.463 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:30.595925Z" 2020-07-17T12:08:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:47.465 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.465 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.465 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.465 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.467 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.467 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.467 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.468 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.468 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:47.468 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.469 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.469 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.469 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.469 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.469 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.470 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.470 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.471 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.471 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.471 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.471 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.472 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.472 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:47.472 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.473 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.473 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.473 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.473 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.473 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.474 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.474 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.475 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.475 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.475 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.476 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.476 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.477 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:47.478 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.478 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.478 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.479 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.479 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.479 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.480 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.481 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.482 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.482 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.482 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.482 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.483 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:47.484 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:47.484 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:47.484 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:47.485 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:47.485 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:47.485 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:47.486 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:47.486 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:47.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.487 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:47.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:47.487 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:47.488 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:47.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:47.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.489 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.489 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:47.489 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:08:47.490 WARNING:rgw_multi.tests:bucket lbjidp-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000004.19.9 target=00000000002.15.10 2020-07-17T12:08:52.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:52.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:52.493 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:52.539 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:52.539 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:52.747 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:08:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:08:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:08:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:08:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.19.9", 2020-07-17T12:08:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:30.393249Z" 2020-07-17T12:08:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:52.789 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:08:52.790 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:08:54.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:54.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:54.533 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:54.575 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:54.575 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:54.775 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:08:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:08:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:08:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:08:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.24.5" 2020-07-17T12:08:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:08:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:08:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.11.9" 2020-07-17T12:08:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:08:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:08:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000009.23.9" 2020-07-17T12:08:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:08:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:08:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:08:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:08:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:08:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:08:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:08:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:08:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:08:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:08:54.784 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:08:54.785 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:08:54.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:54.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:08:54.788 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:08:54.835 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:08:54.835 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:08:55.033 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:08:55.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.033 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:08:55.034 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.034 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:55.034 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.034 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.034 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:55.035 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:55.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.035 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:55.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.035 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.036 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:55.036 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:55.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.037 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:08:55.037 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.037 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:55.037 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.038 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.038 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:08:55.038 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:08:55.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.039 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:08:55.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.039 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.039 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.24.5", 2020-07-17T12:08:55.039 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.921259Z" 2020-07-17T12:08:55.040 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.040 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.040 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:08:55.040 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.041 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:55.041 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.041 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.041 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:55.041 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:55.042 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.042 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:55.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.042 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.044 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:08:55.044 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.044 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:55.044 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.044 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.045 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:08:55.045 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:08:55.045 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.045 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:08:55.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.046 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.046 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.9.10", 2020-07-17T12:08:55.046 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.991893Z" 2020-07-17T12:08:55.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.047 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:08:55.048 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.048 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:55.048 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.048 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.048 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:55.049 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:55.049 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.049 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:55.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.049 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.050 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:55.050 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:55.050 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.050 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.051 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:08:55.051 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.051 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:08:55.051 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.051 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.052 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:08:55.052 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:08:55.052 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.052 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:08:55.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.053 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.053 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000008.21.10", 2020-07-17T12:08:55.053 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.192859Z" 2020-07-17T12:08:55.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.054 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:08:55.054 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.054 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:55.054 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.055 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.055 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:55.055 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:55.055 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.056 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:55.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.056 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.056 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:55.056 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:55.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.057 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.057 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:08:55.057 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.058 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:55.058 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.058 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.058 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:55.059 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:55.059 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.059 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:55.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.060 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.060 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:55.060 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:55.060 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.060 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.061 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:08:55.061 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.061 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:55.061 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.062 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.062 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:55.062 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:55.062 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.062 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:55.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.063 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.063 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:55.063 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:55.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.064 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:08:55.064 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.065 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:55.065 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.066 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.066 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:55.066 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:55.066 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.067 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:55.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.067 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.067 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:55.067 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:55.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:08:55.068 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:08:55.068 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:08:55.069 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:08:55.069 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:08:55.069 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:08:55.069 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:08:55.069 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:08:55.070 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:08:55.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.070 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:08:55.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:08:55.070 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:08:55.070 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:08:55.071 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:08:55.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:08:55.071 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:08:55.072 WARNING:rgw_multi.tests:bucket lbjidp-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000003.11.9 target=00000000002.9.10, shard=5 master=00000000009.23.9 target=00000000008.21.10 2020-07-17T12:09:00.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:00.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:00.074 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:00.126 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:00.126 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:00.324 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:00.325 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:00.326 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.327 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.329 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:09:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.330 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.24.5", 2020-07-17T12:09:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.921259Z" 2020-07-17T12:09:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.332 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:00.333 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:00.334 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.335 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:00.336 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:09:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.337 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.9.10", 2020-07-17T12:09:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.991893Z" 2020-07-17T12:09:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.338 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.339 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.340 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:00.341 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.342 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:09:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:09:00.343 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000008.21.10", 2020-07-17T12:09:00.344 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.192859Z" 2020-07-17T12:09:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.345 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:00.346 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.347 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.349 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.350 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:00.351 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.352 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:00.353 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.354 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.355 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.356 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:00.356 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.357 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:00.357 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.358 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.358 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:00.358 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:00.358 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.359 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:00.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.359 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.359 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:00.359 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:00.365 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.365 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:00.366 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:00.366 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:00.366 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:00.366 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:00.366 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:00.367 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:00.367 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:00.367 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:00.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.367 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:00.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:00.368 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:00.368 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:00.368 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:00.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.369 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:00.369 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:00.369 WARNING:rgw_multi.tests:bucket lbjidp-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000003.11.9 target=00000000002.9.10, shard=5 master=00000000009.23.9 target=00000000008.21.10 2020-07-17T12:09:05.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:05.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:05.373 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:05.424 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:05.424 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:05.638 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:05.639 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.639 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:05.639 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.639 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:05.639 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.639 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.640 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:05.640 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:05.640 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.640 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:05.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.641 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.641 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:05.641 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:05.641 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.642 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.642 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:05.642 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.642 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:05.643 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.643 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.643 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:05.643 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:09:05.643 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.644 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:05.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.644 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.644 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.24.5", 2020-07-17T12:09:05.644 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.921259Z" 2020-07-17T12:09:05.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.645 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:05.646 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.646 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:05.646 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.646 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.646 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:05.647 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:05.647 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.647 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:05.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.647 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.648 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:05.648 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:05.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.649 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:05.649 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.649 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:05.649 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.649 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.650 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:05.650 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:09:05.650 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.650 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:05.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.651 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.651 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.9.10", 2020-07-17T12:09:05.651 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.991893Z" 2020-07-17T12:09:05.651 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.652 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.653 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:05.653 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.654 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:05.654 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.654 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.654 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:05.654 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:05.654 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.655 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:05.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.655 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.655 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:05.656 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:05.656 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.656 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.656 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:05.657 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.657 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:05.657 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.657 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.657 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:09:05.658 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:09:05.658 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.658 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:05.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.658 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.659 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000008.21.10", 2020-07-17T12:09:05.659 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.192859Z" 2020-07-17T12:09:05.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.660 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:05.660 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.660 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:05.660 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.660 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.661 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:05.661 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:05.661 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.661 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:05.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.662 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.662 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:05.662 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:05.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.663 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:05.663 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.663 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:05.663 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.664 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.664 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:05.664 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:05.664 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.665 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:05.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.665 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.665 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:05.665 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:05.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.666 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.666 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.666 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:05.666 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.667 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:05.667 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.667 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.667 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:05.667 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:05.667 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.668 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:05.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.668 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.668 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:05.669 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:05.669 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.669 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.669 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.669 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:05.670 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.671 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:05.671 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.671 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.671 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:05.671 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:05.672 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.672 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:05.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.672 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.673 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:05.673 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:05.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:05.674 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:05.674 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:05.674 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:05.674 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:05.674 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:05.675 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:05.675 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:05.675 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:05.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.675 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:05.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:05.676 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:05.676 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:05.676 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:05.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.677 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.677 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:05.677 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:05.678 WARNING:rgw_multi.tests:bucket lbjidp-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000003.11.9 target=00000000002.9.10, shard=5 master=00000000009.23.9 target=00000000008.21.10 2020-07-17T12:09:10.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:10.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:10.681 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:10.728 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:10.729 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:10.917 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:10.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.917 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:10.917 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.918 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:10.918 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.918 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.918 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:10.918 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:10.919 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.919 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:10.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.919 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.919 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:10.920 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:10.920 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.920 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.921 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:10.921 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.921 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:10.921 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.921 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.922 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:10.922 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:09:10.922 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.922 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:10.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.923 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.923 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.24.5", 2020-07-17T12:09:10.923 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.921259Z" 2020-07-17T12:09:10.923 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.923 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.924 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:10.924 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.924 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:10.925 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.925 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.925 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:10.925 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:10.925 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.926 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:10.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.926 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.926 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:10.926 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:10.927 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.927 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.927 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:10.928 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.928 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:10.928 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.928 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.928 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:10.928 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:09:10.929 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.929 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:10.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.929 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.931 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.9.10", 2020-07-17T12:09:10.931 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.991893Z" 2020-07-17T12:09:10.931 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.931 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.932 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.932 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:10.932 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.932 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:10.932 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.932 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.933 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:10.933 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:10.933 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.933 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:10.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.934 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.934 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:10.934 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:10.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.935 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.935 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:10.935 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.935 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:10.936 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.936 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.936 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:09:10.936 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:09:10.936 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.937 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:10.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.937 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.937 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000009.23.9", 2020-07-17T12:09:10.937 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.192859Z" 2020-07-17T12:09:10.938 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.938 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.938 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:10.939 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.939 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:10.939 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.939 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.939 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:10.940 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:10.940 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.940 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:10.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.941 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.941 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:10.941 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:10.941 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.941 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.942 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.942 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:10.942 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.942 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:10.942 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.943 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.943 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:10.943 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:10.943 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.943 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:10.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.944 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.944 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:10.944 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:10.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.945 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:10.945 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.945 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:10.946 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.946 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.946 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:10.946 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:10.946 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.947 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:10.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.947 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.947 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:10.947 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:10.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.948 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:10.948 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.949 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:10.949 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.950 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.950 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:10.950 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:10.950 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.951 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:10.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.951 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.951 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:10.951 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:10.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.952 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:10.952 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:10.952 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:10.952 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:10.953 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:10.953 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:10.953 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:10.953 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:10.953 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:10.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.954 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:10.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:10.954 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:10.954 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:10.955 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:10.955 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.955 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.955 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:10.955 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:10.956 WARNING:rgw_multi.tests:bucket lbjidp-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000003.11.9 target=00000000002.9.10 2020-07-17T12:09:15.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:15.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:15.960 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:16.006 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:16.006 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:16.211 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:16.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.212 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:16.212 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.212 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:16.212 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.213 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.213 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:16.213 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:16.213 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.214 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:16.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.214 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.214 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:16.214 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.215 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.215 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.215 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:16.216 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.216 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:16.216 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.216 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.216 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:16.217 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:09:16.217 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.217 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:16.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.217 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.218 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.24.5", 2020-07-17T12:09:16.218 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.921259Z" 2020-07-17T12:09:16.218 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.218 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.219 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:16.219 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.219 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:16.219 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.220 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.220 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:16.220 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:16.220 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.221 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:16.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.221 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.221 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:16.222 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.222 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.222 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.222 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:16.223 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.223 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:16.223 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.223 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.223 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:16.224 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:09:16.224 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.224 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:16.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.225 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.225 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.11.9", 2020-07-17T12:09:16.225 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.991893Z" 2020-07-17T12:09:16.225 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.225 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.226 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:16.226 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.226 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:16.226 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.227 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.227 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:16.227 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:16.227 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.228 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:16.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.228 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.228 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:16.228 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.228 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.229 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:16.229 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.230 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:16.230 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.230 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.230 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:09:16.230 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:09:16.230 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.231 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:16.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.231 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.231 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000009.23.9", 2020-07-17T12:09:16.232 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.192859Z" 2020-07-17T12:09:16.232 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.232 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.232 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:16.233 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.233 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:16.233 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.233 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.233 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:16.233 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:16.234 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.234 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:16.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.234 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.235 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:16.235 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.235 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.235 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.236 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:16.236 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.236 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:16.236 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.236 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.236 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:16.237 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:16.237 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.237 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:16.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.238 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.239 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:16.239 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.239 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.239 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.240 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.240 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:16.240 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.240 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:16.240 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.241 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.241 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:16.241 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:16.241 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.242 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:16.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.242 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.242 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:16.242 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.243 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.243 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.243 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:16.243 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.244 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:16.245 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.245 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.245 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:16.245 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:16.245 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.246 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:16.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.246 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.246 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:16.247 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.247 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.247 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:16.248 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:16.248 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:16.248 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:16.248 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:16.248 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:16.248 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:16.249 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:16.249 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:16.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.249 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:16.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:16.250 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:16.250 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:16.250 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:16.251 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:16.251 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:09:16.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:16.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:16.253 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:16.296 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:16.296 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:16.506 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:09:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:09:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.24.5" 2020-07-17T12:09:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.11.9" 2020-07-17T12:09:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000009.23.9" 2020-07-17T12:09:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:16.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.514 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:16.514 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:16.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.515 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:16.515 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:16.515 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.515 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:09:16.515 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:09:16.516 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:09:16.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:16.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:16.518 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:16.603 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:16.603 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:16.818 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:16.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.818 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:16.818 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.819 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:16.819 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.819 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.819 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:16.819 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:16.820 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.820 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.822 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:16.822 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.823 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:16.823 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.823 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.823 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:16.823 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:09:16.823 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.824 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:16.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.824 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.824 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.24.5", 2020-07-17T12:09:16.825 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:52.921259Z" 2020-07-17T12:09:16.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.826 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:16.826 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.826 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:16.826 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.826 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.827 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:16.827 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:16.827 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.827 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:16.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.828 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.828 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:16.828 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.829 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:16.829 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.829 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:09:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.11.9", 2020-07-17T12:09:16.832 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:52.991893Z" 2020-07-17T12:09:16.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.833 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:16.833 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.833 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:16.833 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.833 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.833 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:16.834 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:16.834 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.834 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:16.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.835 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.835 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:16.835 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.835 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.836 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:16.836 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.837 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:16.837 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.837 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.837 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:16.837 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:09:16.837 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.838 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:16.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.838 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.838 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000009.23.9", 2020-07-17T12:09:16.839 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.192859Z" 2020-07-17T12:09:16.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.839 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:16.840 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.840 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:16.840 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.840 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.840 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:16.841 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:16.841 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.841 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:16.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.842 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.842 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:16.842 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.843 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.843 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.844 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:16.844 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.844 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:16.844 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.844 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.844 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:16.845 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:16.845 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.845 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:16.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.846 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.846 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:16.846 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.847 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:16.847 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.847 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:16.847 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.848 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.848 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:16.848 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:16.848 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.848 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:16.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.849 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.849 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:16.849 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.850 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:16.850 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.851 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:16.852 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.852 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.852 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:16.852 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:16.852 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.853 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:16.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.853 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.853 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:16.853 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:16.854 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:16.855 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:16.855 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:16.855 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:16.858 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:16.859 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:09:16.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:16.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:16.860 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:16.932 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:16.933 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:17.157 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:09:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:09:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.24.5" 2020-07-17T12:09:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.11.9" 2020-07-17T12:09:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000009.23.9" 2020-07-17T12:09:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:09:17.167 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:09:17.168 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:09:17.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:17.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:17.171 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:17.215 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:17.216 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:17.418 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:17.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.418 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:17.419 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.419 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:17.419 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.419 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.420 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.420 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.420 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.420 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.421 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.421 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:17.421 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:17.421 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.423 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:17.423 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.423 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.423 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.424 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.424 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.424 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.425 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.425 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.24.5", 2020-07-17T12:09:17.425 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.921259Z" 2020-07-17T12:09:17.426 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.426 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.427 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:17.427 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.427 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:17.427 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.428 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.428 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.428 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.428 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:17.430 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:17.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.431 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:17.431 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.431 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:17.431 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.431 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.432 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.432 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.432 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.433 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.433 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.434 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.11.9", 2020-07-17T12:09:17.434 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.991893Z" 2020-07-17T12:09:17.434 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.435 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.435 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:17.436 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.436 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:17.436 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.436 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.436 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.437 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.437 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.437 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.438 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.438 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:17.438 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:17.438 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.439 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.439 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.439 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:17.439 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.440 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:17.440 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.440 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.440 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.440 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.441 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.441 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.442 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.442 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000009.23.9", 2020-07-17T12:09:17.442 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.192859Z" 2020-07-17T12:09:17.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.444 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:17.444 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.444 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.446 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.446 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.448 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.448 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.448 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:17.448 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.449 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:17.449 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.449 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.449 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.450 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.450 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.451 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.451 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.453 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:17.453 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.453 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:17.453 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.454 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.454 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.454 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.454 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:17.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.456 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:17.457 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.457 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:17.457 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.457 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.458 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.481 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.481 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.481 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.482 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.482 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:17.482 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:17.483 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.483 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.484 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:17.484 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:17.484 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:17.484 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:17.485 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:17.485 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:17.485 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:17.485 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:17.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.486 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:17.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.488 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:17.488 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:17.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:17.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.489 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:17.490 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:09:17.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:17.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:17.492 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:17.535 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:17.535 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:17.759 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:09:17.760 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:09:17.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.760 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:17.761 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:17.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.762 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:17.762 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.15.9" 2020-07-17T12:09:17.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.762 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:17.763 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:17.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.764 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:17.764 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000010.27.5" 2020-07-17T12:09:17.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.764 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:17.765 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:17.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.766 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:17.766 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000007.23.9" 2020-07-17T12:09:17.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.766 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:17.767 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:17.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.767 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:17.767 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:17.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.768 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:17.768 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:17.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.769 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:17.770 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:17.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:17.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:17.771 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:17.771 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:17.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:17.772 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:09:17.772 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:09:17.773 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-16 2020-07-17T12:09:17.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:17.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:17.776 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:17.821 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:17.822 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:18.021 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.023 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.024 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:18.026 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:09:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.15.9", 2020-07-17T12:09:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:52.800749Z" 2020-07-17T12:09:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:18.033 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.042 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:09:18.043 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.044 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.27.5", 2020-07-17T12:09:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.113092Z" 2020-07-17T12:09:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.046 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.048 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.050 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.051 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:09:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:18.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000007.23.9", 2020-07-17T12:09:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.192859Z" 2020-07-17T12:09:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.053 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:18.054 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.055 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.056 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.058 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.059 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.060 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.061 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.063 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.064 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:18.066 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.067 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.069 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.070 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.071 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.072 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.074 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:18.075 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-16 2020-07-17T12:09:18.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:18.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:18.077 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:18.118 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:18.118 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:18.336 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:09:18.336 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:09:18.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.337 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:18.337 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:18.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.338 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:18.338 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.15.9" 2020-07-17T12:09:18.338 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.338 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.338 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:18.339 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:18.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.339 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.339 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:18.339 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000010.27.5" 2020-07-17T12:09:18.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.340 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:18.340 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:18.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.341 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:18.341 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000007.23.9" 2020-07-17T12:09:18.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.341 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.341 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:18.342 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:18.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.342 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:18.343 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:18.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.343 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.343 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:18.344 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:18.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.344 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.344 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:18.344 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:18.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:18.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:18.345 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:18.345 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:18.345 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:18.345 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:09:18.346 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:09:18.347 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-16 2020-07-17T12:09:18.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:18.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:18.348 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:18.393 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:18.393 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:18.594 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:18.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.594 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:18.595 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.595 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.595 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.595 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.596 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.596 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.596 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.596 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.597 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.597 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.597 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.598 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.598 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:18.598 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.599 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:18.599 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.599 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.599 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:18.599 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:09:18.600 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.600 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:18.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.600 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.600 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.15.9", 2020-07-17T12:09:18.601 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:52.800749Z" 2020-07-17T12:09:18.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.602 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:18.602 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.602 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.602 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.602 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.603 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.603 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.603 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.603 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.604 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.604 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.604 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.605 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:18.605 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.605 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:18.605 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.606 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.606 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:18.606 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:09:18.606 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.607 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:18.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.607 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.607 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.27.5", 2020-07-17T12:09:18.608 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.113092Z" 2020-07-17T12:09:18.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.609 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:18.609 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.609 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.609 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.609 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.612 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.612 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.612 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.613 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.613 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.613 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.614 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.615 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.615 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.615 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:18.616 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.616 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:18.616 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.616 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.616 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:18.617 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:09:18.617 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.617 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:18.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.618 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.618 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000007.23.9", 2020-07-17T12:09:18.618 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.192859Z" 2020-07-17T12:09:18.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.619 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:18.619 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.619 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.620 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.620 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.620 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.620 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.620 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.621 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.621 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.621 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.622 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.623 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:18.623 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.623 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.623 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.624 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.624 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.625 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.625 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.625 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.626 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.626 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.626 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.627 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:18.627 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.627 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.628 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.628 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.628 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.628 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.628 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.629 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.629 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.629 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.630 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.631 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:18.631 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.632 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.632 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.632 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.632 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.633 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.633 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.633 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.634 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.634 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.634 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.635 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:18.635 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:18.635 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:18.636 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:18.636 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:18.636 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:18.636 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:18.636 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:18.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.637 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:18.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.637 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:18.637 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:18.637 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:18.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.638 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:18.639 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-16 2020-07-17T12:09:18.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:18.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:18.640 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:18.890 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:09:18.890 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:09:18.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.891 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:18.891 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:18.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.892 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:18.892 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.11.9" 2020-07-17T12:09:18.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.893 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:18.893 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:18.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.894 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:18.894 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.16.9" 2020-07-17T12:09:18.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.895 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:18.895 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:18.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.896 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:18.896 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000014.38.5" 2020-07-17T12:09:18.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.897 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:18.897 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:18.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.898 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:18.898 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:18.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.899 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:18.899 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:18.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.900 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:18.900 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:18.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:18.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:18.900 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:18.900 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:18.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:18.901 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:09:18.901 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:09:18.902 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-16 2020-07-17T12:09:18.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:18.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:18.904 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:18.987 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:18.987 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:19.189 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:19.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.190 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:19.190 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.190 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:19.191 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.191 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.192 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:19.192 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:19.192 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.193 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:19.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.194 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.194 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:19.194 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.196 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:19.196 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.197 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:19.197 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.197 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.198 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:19.198 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:09:19.198 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.199 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:19.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.200 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.200 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.11.9", 2020-07-17T12:09:19.201 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:52.800749Z" 2020-07-17T12:09:19.201 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.201 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.202 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.204 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.204 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:19.205 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:19.205 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.206 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:19.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.206 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.207 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:19.207 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.208 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.209 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:19.209 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.209 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:19.210 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.210 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.211 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:19.211 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:09:19.212 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.212 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:19.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.213 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.213 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.16.9", 2020-07-17T12:09:19.214 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:52.991893Z" 2020-07-17T12:09:19.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.215 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:19.216 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.216 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:19.216 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.217 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.217 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:19.217 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:19.218 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.220 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:19.220 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.220 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.222 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:19.222 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.222 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:19.223 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.223 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.223 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:19.224 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:09:19.224 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.225 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:19.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.225 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.226 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000014.38.5", 2020-07-17T12:09:19.226 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.303618Z" 2020-07-17T12:09:19.226 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.227 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.228 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:19.228 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.228 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:19.229 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.229 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.230 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:19.230 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:19.230 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.231 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:19.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.232 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.232 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:19.233 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.233 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.233 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.234 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:19.235 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.235 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:19.235 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.236 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.236 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:19.236 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:19.237 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.237 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:19.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.239 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.239 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:19.239 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.240 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.240 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.241 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:19.241 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.242 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:19.242 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.242 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.243 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:19.243 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:19.243 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.244 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:19.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.245 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.245 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:19.245 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.246 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.246 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.247 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:19.247 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.249 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:19.249 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.250 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.250 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:19.250 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:19.251 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.251 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:19.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.252 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.252 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:19.253 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.253 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.253 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.254 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:19.255 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:19.255 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:19.255 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:19.256 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:19.256 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:19.256 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:19.257 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:19.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.258 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:19.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.258 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:19.259 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:19.259 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.259 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.260 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:19.261 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-16 2020-07-17T12:09:19.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:19.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:19.264 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:19.330 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:19.330 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:19.545 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:09:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:09:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.11.9" 2020-07-17T12:09:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.16.9" 2020-07-17T12:09:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000014.38.5" 2020-07-17T12:09:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:09:19.554 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:09:19.555 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-16 2020-07-17T12:09:19.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:19.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:19.557 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:19.603 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:19.604 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:19.810 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:19.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.811 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:19.811 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.811 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:19.811 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.811 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.812 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:19.812 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:19.812 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.812 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:19.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.813 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.813 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:19.813 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.813 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.813 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.814 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:19.814 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.814 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:19.815 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.815 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.815 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:19.815 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:09:19.815 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.816 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:19.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.816 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.816 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.11.9", 2020-07-17T12:09:19.816 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.800749Z" 2020-07-17T12:09:19.817 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.817 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.817 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:19.818 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.818 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:19.818 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.818 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.818 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:19.819 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:19.819 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.819 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:19.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.819 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.820 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:19.820 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.821 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:19.821 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.821 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:19.821 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.821 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.822 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:19.822 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:09:19.822 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.822 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:19.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.823 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.823 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.16.9", 2020-07-17T12:09:19.823 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:52.991893Z" 2020-07-17T12:09:19.823 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.823 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.824 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:19.824 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.824 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:19.825 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.825 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.825 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:19.825 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:19.825 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.830 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:19.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.830 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.830 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:19.831 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.832 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:19.832 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.832 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:19.832 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.832 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.832 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:09:19.833 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:09:19.833 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.833 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:19.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.834 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.834 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000014.38.5", 2020-07-17T12:09:19.834 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.303618Z" 2020-07-17T12:09:19.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.835 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:19.835 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.836 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:19.836 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.836 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.836 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:19.837 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:19.837 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.837 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:19.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.837 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.838 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:19.838 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.839 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:19.839 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.839 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:19.839 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.840 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.840 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:19.840 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:19.840 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.840 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:19.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.841 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.841 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:19.841 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.842 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.842 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:19.842 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.842 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:19.843 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.843 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.843 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:19.843 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:19.843 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.844 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:19.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.844 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.844 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:19.845 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.846 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:19.846 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.847 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:19.847 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.847 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.847 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:19.848 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:19.848 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.848 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:19.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.849 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.849 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:19.849 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:19.850 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:19.850 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:19.850 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:19.850 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:19.851 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:19.851 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:19.851 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:19.851 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:19.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.851 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:19.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:19.852 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:19.852 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:19.852 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:19.852 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:19.853 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:19.853 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-16 2020-07-17T12:09:19.854 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-16 zones={test-zone2, test-zone1} 2020-07-17T12:09:19.865 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:09:19.872 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:09:19.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:09:19.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:09:19.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:09:19.874 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:09:19.880 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:09:19.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:09:19.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:09:19.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:09:19.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:19.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:19.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:20.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:20.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:20.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:20.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:09:20.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:09:20.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:09:20.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:09:20.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:09:20.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:09:20.889 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-16 zones={test-zone2, test-zone1} 2020-07-17T12:09:20.890 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-16 zones={test-zone3, test-zone1} 2020-07-17T12:09:20.898 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:09:20.903 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:09:20.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:09:20.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:09:20.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:09:20.905 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:09:20.909 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:09:20.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:09:20.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:09:20.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:09:20.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:20.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:20.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:21.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:21.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:21.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:21.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:09:21.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:09:21.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:09:21.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:09:21.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:09:21.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:09:21.885 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-16 zones={test-zone3, test-zone1} 2020-07-17T12:09:21.886 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-16 zones={test-zone3, test-zone2} 2020-07-17T12:09:21.897 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:09:21.903 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:09:21.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:09:21.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:09:21.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:09:21.905 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:09:21.913 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:09:21.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:09:21.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:09:21.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:09:21.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:21.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:22.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:22.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:22.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:22.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:22.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:09:22.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:09:22.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:09:22.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:09:22.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:09:22.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:09:22.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-16 zones={test-zone3, test-zone2} 2020-07-17T12:09:22.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:22.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:22.905 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:23.162 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:09:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:09:23.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:23.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:23.164 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.20.5" 2020-07-17T12:09:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.165 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.167 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:23.167 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.10.9" 2020-07-17T12:09:23.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.168 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:23.168 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:23.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.169 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000006.17.10" 2020-07-17T12:09:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:23.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:23.172 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.181 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:23.182 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:23.183 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:09:23.184 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:09:23.185 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:09:23.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:23.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:23.187 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:23.234 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:23.235 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:23.449 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:23.450 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.450 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:23.450 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.450 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:23.451 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.451 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.451 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:23.451 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:23.451 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.452 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:23.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.452 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.452 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:23.453 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:23.453 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.453 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.454 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:23.454 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.454 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:23.454 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.454 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.455 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:23.455 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:09:23.455 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.455 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:23.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.456 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.456 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.20.5", 2020-07-17T12:09:23.456 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.482799Z" 2020-07-17T12:09:23.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.457 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.457 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:23.457 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.457 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:23.458 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.458 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.458 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:23.458 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:23.458 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.459 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:23.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.459 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.460 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:23.460 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:23.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.461 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:23.461 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.461 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:23.461 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.461 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.462 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:23.462 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:23.462 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.462 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:23.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.463 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.463 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.10.9", 2020-07-17T12:09:23.463 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:23.463 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.463 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:23.464 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.466 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:23.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.466 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.468 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.469 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:09:23.469 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:23.469 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.469 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.13.9", 2020-07-17T12:09:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:08.162845Z" 2020-07-17T12:09:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.470 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.471 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:23.473 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.474 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.474 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:23.475 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.475 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:23.475 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.475 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.475 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.476 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:23.488 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:23.489 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000006.17.10 target=00000000004.13.9 2020-07-17T12:09:28.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:28.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:28.492 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:28.542 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:28.552 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:28.757 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:28.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.758 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:28.759 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.759 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:28.759 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.760 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.760 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:28.760 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:28.760 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.761 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:28.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.762 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.762 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:28.762 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:28.763 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.763 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:09:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.20.5", 2020-07-17T12:09:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.482799Z" 2020-07-17T12:09:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:28.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.771 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.771 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:28.771 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:28.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.772 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.772 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:28.772 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.10.9", 2020-07-17T12:09:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:28.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.779 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:28.780 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.780 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:28.780 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.780 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.780 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:09:28.781 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:28.781 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.781 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:28.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.782 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.782 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.13.9", 2020-07-17T12:09:28.782 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:08.162845Z" 2020-07-17T12:09:28.782 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.782 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.783 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:28.783 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.783 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:28.784 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.784 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.784 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:28.784 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:28.784 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.785 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:28.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.785 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.785 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:28.786 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:28.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.787 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:28.787 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.787 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:28.787 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.787 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.788 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:28.788 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:28.788 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.788 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:28.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.789 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.789 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:28.789 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:28.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.790 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.790 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:28.790 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.791 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:28.791 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.791 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.791 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:28.791 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:28.792 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.792 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:28.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.792 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.792 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:28.793 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:28.793 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.793 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.794 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:28.794 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.795 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:28.795 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.795 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.796 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:28.796 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:28.796 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.796 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:28.797 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:28.798 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:28.799 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:28.800 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.801 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:28.802 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:28.802 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000006.17.10 target=00000000004.13.9 2020-07-17T12:09:33.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:33.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:33.808 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:33.868 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:33.869 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:34.062 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:09:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.068 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.20.5", 2020-07-17T12:09:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.482799Z" 2020-07-17T12:09:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:34.074 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.10.9", 2020-07-17T12:09:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:34.082 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:09:34.083 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.084 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.085 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.17.10", 2020-07-17T12:09:34.085 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:09:34.085 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.085 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.086 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:34.087 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.087 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:34.087 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.087 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.087 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:34.088 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:34.088 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.088 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:34.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.089 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.089 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:34.089 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.090 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:34.090 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.090 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:34.091 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.091 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.091 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:34.091 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:34.091 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.092 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:34.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.092 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.093 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:34.093 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.095 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:34.095 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.095 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:34.095 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.096 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.096 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:34.096 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:34.096 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.096 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:34.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.097 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.097 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:34.097 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.097 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.098 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:34.098 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.100 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:34.100 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.100 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.100 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:34.100 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:34.101 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.101 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:34.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.101 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.102 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:34.102 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:34.103 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:34.104 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:34.104 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:34.104 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:34.104 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:34.104 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:34.105 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:34.105 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:34.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.105 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:34.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:34.106 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:34.106 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:34.106 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:34.107 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:34.107 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:09:34.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:34.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:34.109 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:34.157 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:34.358 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:09:34.358 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:09:34.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.359 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:34.359 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:34.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.359 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:34.359 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.20.5" 2020-07-17T12:09:34.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.360 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:34.360 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:34.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.361 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:34.361 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.10.9" 2020-07-17T12:09:34.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.362 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:34.362 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:34.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.362 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:34.363 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000007.19.9" 2020-07-17T12:09:34.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.363 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:34.363 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:34.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.364 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:34.364 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:34.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.365 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:34.365 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:34.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.365 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:34.366 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:34.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.366 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:34.366 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:34.367 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.367 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:09:34.367 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:09:34.368 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:09:34.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:34.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:34.370 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:34.454 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:34.454 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:34.685 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:34.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.686 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:34.686 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.687 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:34.687 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.687 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.688 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:34.688 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:34.688 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.689 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:34.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.689 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.690 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:34.690 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.692 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:34.692 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.692 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:34.693 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.693 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.693 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:34.694 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:09:34.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.695 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:34.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.695 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.695 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.20.5", 2020-07-17T12:09:34.696 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.482799Z" 2020-07-17T12:09:34.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.697 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:34.706 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.706 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:34.706 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.707 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.707 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:34.707 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:34.707 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:34.709 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.710 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:34.710 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.710 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:34.710 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.10.9", 2020-07-17T12:09:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:34.714 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.714 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:34.714 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.714 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.714 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:34.714 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:34.715 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.715 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:34.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.715 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000006.17.10", 2020-07-17T12:09:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:09:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.721 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.721 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:34.721 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:34.721 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.722 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:34.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.722 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.722 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:34.723 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.724 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:34.724 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.725 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:34.725 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.725 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.726 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:34.726 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:34.726 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.727 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:34.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.728 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.728 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:34.728 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.729 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.730 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:34.730 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.730 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:34.731 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.731 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.731 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:34.732 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:34.732 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.733 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:34.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.733 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.734 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:34.734 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.734 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.736 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:34.737 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.739 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:34.739 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.739 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.740 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:34.740 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:34.741 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.741 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:34.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.742 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.743 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:34.743 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.743 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:34.745 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:34.745 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:34.746 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:34.746 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:34.746 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:34.747 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:34.747 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:34.748 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:34.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.748 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:34.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:34.749 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:34.750 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:34.750 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:34.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:34.752 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:34.752 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.19.9 target=00000000006.17.10 2020-07-17T12:09:39.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:39.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:39.756 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:39.801 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:39.802 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:40.004 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:40.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.005 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:40.005 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.005 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:40.005 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.006 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.006 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:40.006 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:40.006 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.007 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:40.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.007 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.007 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:40.008 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:40.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.009 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:40.009 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.009 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:40.010 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.010 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.010 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:40.010 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:09:40.010 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.011 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:40.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.012 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.012 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.20.5", 2020-07-17T12:09:40.012 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.482799Z" 2020-07-17T12:09:40.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.013 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:40.013 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.013 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:40.014 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.015 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.015 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:40.015 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:40.015 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.016 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:40.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.016 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.016 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:40.017 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:40.017 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.017 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.018 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:40.018 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.018 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:40.018 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.019 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.019 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:40.019 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:40.019 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.10.9", 2020-07-17T12:09:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000006.17.10", 2020-07-17T12:09:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:09:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:40.051 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:40.051 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.19.9 target=00000000006.17.10 2020-07-17T12:09:45.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:45.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:45.055 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:45.101 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:45.102 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:45.292 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:09:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.20.5", 2020-07-17T12:09:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.482799Z" 2020-07-17T12:09:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.10.9", 2020-07-17T12:09:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000006.17.10", 2020-07-17T12:09:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:09:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:45.366 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:45.367 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.19.9 target=00000000006.17.10 2020-07-17T12:09:50.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:50.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:50.370 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:50.413 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:50.414 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:50.618 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:50.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.620 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:50.620 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.620 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:50.620 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.621 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.621 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:50.621 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:50.622 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.622 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:50.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.624 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:50.625 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.625 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:50.625 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.625 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.626 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:50.626 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:09:50.626 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.627 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:50.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.627 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.627 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.20.5", 2020-07-17T12:09:50.628 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.482799Z" 2020-07-17T12:09:50.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.639 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:50.645 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:50.646 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.646 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:50.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.647 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.647 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:50.648 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:50.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.649 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:50.649 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.650 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:50.650 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.650 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.651 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:50.651 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:50.651 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.652 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:50.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.653 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.653 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.10.9", 2020-07-17T12:09:50.653 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:50.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.655 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:50.656 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.656 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:50.656 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.657 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.657 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:50.657 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:50.658 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.658 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:50.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.659 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.659 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:50.659 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:50.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.661 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:50.661 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.661 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:50.661 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.662 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.662 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:50.662 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:50.662 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.663 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:50.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.663 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.664 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000006.17.10", 2020-07-17T12:09:50.664 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:09:50.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.665 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:50.665 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:50.685 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:50.685 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.19.9 target=00000000006.17.10 2020-07-17T12:09:55.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:55.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:55.688 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:55.733 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:55.733 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:55.941 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:09:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.20.5", 2020-07-17T12:09:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.482799Z" 2020-07-17T12:09:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.10.9", 2020-07-17T12:09:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.311 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:56.311 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.311 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:56.311 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.312 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.312 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:56.312 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:56.312 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.313 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:56.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.313 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.313 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:56.313 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.314 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:56.315 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.315 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:56.315 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.315 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.315 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:56.316 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:56.316 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.316 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:56.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.316 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.317 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000007.19.9", 2020-07-17T12:09:56.317 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:09:56.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.318 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:56.318 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.318 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:56.318 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.319 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.319 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:56.319 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:56.319 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.320 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:56.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.321 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.321 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:56.321 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.321 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.322 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:56.322 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.322 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:56.323 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.323 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.323 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:56.323 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:56.323 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.324 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:56.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.324 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.324 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:56.325 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.326 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:56.326 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.326 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:56.326 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.326 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.327 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:56.327 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:56.327 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.327 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:56.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.328 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.328 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:56.328 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.329 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.329 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:56.329 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.330 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:56.331 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.331 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.331 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:56.331 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:56.332 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.332 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:56.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.332 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.333 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:56.333 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.333 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.333 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.334 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:56.334 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:56.334 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:56.334 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:56.335 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:56.335 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:56.335 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:56.335 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:56.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.335 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:56.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.336 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:56.336 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:56.336 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.337 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:56.337 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:09:56.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:56.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:56.339 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:56.413 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:56.413 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:56.646 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:09:56.647 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:09:56.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.647 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:56.647 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:56.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.648 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:56.648 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.20.5" 2020-07-17T12:09:56.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.649 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:56.649 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:56.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.650 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:56.650 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.10.9" 2020-07-17T12:09:56.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.650 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:56.651 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:56.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.651 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:56.651 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000007.19.9" 2020-07-17T12:09:56.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.652 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:56.652 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:56.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.653 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:56.653 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:56.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.653 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:56.653 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:56.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.654 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:56.654 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:56.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:56.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:56.655 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:56.655 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:56.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:56.655 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:09:56.655 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:09:56.656 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:09:56.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:56.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:56.658 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:56.703 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:56.704 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:56.905 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:56.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.20.5", 2020-07-17T12:09:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.482799Z" 2020-07-17T12:09:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.915 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.915 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:56.915 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.916 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:56.916 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.916 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:56.916 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.917 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.917 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.917 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.917 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.917 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.918 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.918 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.10.9", 2020-07-17T12:09:56.918 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:56.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.919 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.919 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:56.919 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.919 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:56.920 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.920 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.920 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.921 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.921 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.922 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.922 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.922 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:56.923 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.923 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.923 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.924 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:56.924 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.924 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:56.924 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.924 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.924 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.925 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.925 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.925 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.926 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.926 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000007.19.9", 2020-07-17T12:09:56.926 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:09:56.926 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.926 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.927 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:56.927 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.928 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:56.928 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.928 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.928 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.929 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.929 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.929 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.930 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.930 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:56.930 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.930 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.930 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.931 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:56.931 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.931 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:56.931 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.932 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.932 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.932 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.932 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.932 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.933 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.933 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:56.933 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.933 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.934 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:56.934 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.934 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:56.935 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.935 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.935 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.935 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.935 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.936 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.936 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.937 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:56.937 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.937 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.938 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.939 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:56.940 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.940 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:56.940 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.940 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.941 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.969 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.969 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.970 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.970 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.970 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:56.971 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.971 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.971 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:56.972 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:56.972 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:56.972 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:56.972 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:56.972 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:56.972 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:56.973 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:56.973 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:56.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.973 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:56.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.974 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.975 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:56.975 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:56.976 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:09:56.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:56.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:56.979 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:57.059 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:57.060 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:57.290 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:09:57.291 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:09:57.291 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.292 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:57.292 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.292 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.8.9" 2020-07-17T12:09:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.293 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000010.26.5" 2020-07-17T12:09:57.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.296 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:57.296 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000009.25.9" 2020-07-17T12:09:57.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.296 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.296 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:57.297 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.297 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:57.297 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.298 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.298 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:57.298 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.299 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:57.299 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.299 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:57.300 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.300 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:57.300 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:09:57.300 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:09:57.301 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:09:57.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:57.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:57.303 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:57.568 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KKBX3XvnfdY4xOQw7oE4r.tBCwfF.YU", 2020-07-17T12:09:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.8.9", 2020-07-17T12:09:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.369745Z" 2020-07-17T12:09:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.26.5", 2020-07-17T12:09:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.790083Z" 2020-07-17T12:09:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000009.25.9", 2020-07-17T12:09:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:09:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:57.616 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:57.617 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:09:57.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:57.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:57.618 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:57.661 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:57.661 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:57.889 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:09:57.889 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:09:57.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.890 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:57.890 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.891 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:57.891 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.8.9" 2020-07-17T12:09:57.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.891 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:57.892 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.892 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:57.892 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000010.26.5" 2020-07-17T12:09:57.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.893 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:57.893 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.894 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:57.894 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000009.25.9" 2020-07-17T12:09:57.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.894 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:57.895 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.895 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:57.895 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.896 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:57.896 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.897 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:57.897 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:57.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:57.898 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:57.898 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:09:57.898 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:57.898 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:09:57.899 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:09:57.900 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:09:57.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:57.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:57.901 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:58.148 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:58.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.149 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:58.149 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.149 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.149 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.150 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.150 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.150 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.150 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.151 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.151 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.151 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.151 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.152 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:58.152 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.153 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:58.153 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.153 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.153 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:58.153 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KKBX3XvnfdY4xOQw7oE4r.tBCwfF.YU", 2020-07-17T12:09:58.153 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.154 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:58.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.154 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.154 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.8.9", 2020-07-17T12:09:58.155 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.369745Z" 2020-07-17T12:09:58.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.156 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:58.156 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.156 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.156 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.156 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.156 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.157 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.157 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.157 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.158 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.158 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.158 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.159 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:58.159 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.159 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:58.159 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.160 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.160 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:58.160 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:58.160 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.160 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:58.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.161 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.161 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.26.5", 2020-07-17T12:09:58.161 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.785979Z" 2020-07-17T12:09:58.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.162 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:58.162 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.162 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.163 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.163 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.163 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.163 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.163 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.164 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.164 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.164 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.164 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.165 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.165 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.165 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:58.166 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.166 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:58.166 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.166 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.166 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:58.166 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:58.167 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.167 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:58.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.167 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.167 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000009.25.9", 2020-07-17T12:09:58.168 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:09:58.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.169 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:58.169 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.169 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.169 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.169 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.169 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.170 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.170 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.170 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.171 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.171 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.171 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.172 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:58.172 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.172 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.172 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.172 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.173 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.173 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.173 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.173 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.174 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.174 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.174 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.175 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.176 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.176 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.176 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.176 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.176 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.177 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.177 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.177 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.177 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.178 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:58.178 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.179 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.180 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.180 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.180 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.180 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.180 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.181 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.181 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.181 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.182 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.182 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:58.183 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.183 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.183 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.183 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.183 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.183 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.184 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.184 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.184 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.185 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.185 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.185 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:58.186 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:09:58.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:58.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:58.188 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:58.262 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:58.263 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:58.483 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:09:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:09:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.483 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:58.484 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.8.9" 2020-07-17T12:09:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:58.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.16.9" 2020-07-17T12:09:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:58.487 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000014.36.5" 2020-07-17T12:09:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:58.491 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:58.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.492 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:58.492 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:58.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.492 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:09:58.493 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:09:58.494 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-17 2020-07-17T12:09:58.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:58.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:58.496 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:58.581 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:58.782 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:09:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KKBX3XvnfdY4xOQw7oE4r.tBCwfF.YU", 2020-07-17T12:09:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.8.9", 2020-07-17T12:09:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.369745Z" 2020-07-17T12:09:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.16.9", 2020-07-17T12:09:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:09:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:09:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:09:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000014.36.5", 2020-07-17T12:09:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.878502Z" 2020-07-17T12:09:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:09:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:09:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:09:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:09:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:09:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:09:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:09:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:09:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:09:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:09:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:09:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:58.847 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:09:58.848 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-17 2020-07-17T12:09:58.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:58.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:58.851 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:58.917 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:58.917 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:59.136 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:09:59.137 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:09:59.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.137 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:09:59.137 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:59.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.138 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:09:59.138 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.8.9" 2020-07-17T12:09:59.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.139 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:09:59.139 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:59.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.140 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:09:59.140 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.16.9" 2020-07-17T12:09:59.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.140 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:09:59.140 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:59.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.141 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:09:59.141 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000014.36.5" 2020-07-17T12:09:59.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.142 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:09:59.142 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:59.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.143 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:09:59.143 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:59.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.143 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:09:59.143 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:59.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.144 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:09:59.144 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:59.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:09:59.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:09:59.145 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:09:59.145 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:09:59.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:09:59.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:09:59.146 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:09:59.147 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-17 2020-07-17T12:09:59.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:59.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:09:59.149 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:09:59.194 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:09:59.195 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:09:59.405 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:09:59.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.406 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:09:59.406 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.406 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:59.406 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.406 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.407 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:59.407 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:59.407 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.407 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:59.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.408 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.408 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:59.408 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:59.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.409 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:09:59.410 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.410 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:59.410 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.410 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.411 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:09:59.411 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "KKBX3XvnfdY4xOQw7oE4r.tBCwfF.YU", 2020-07-17T12:09:59.411 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.411 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:59.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.412 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.412 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.8.9", 2020-07-17T12:09:59.412 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.369745Z" 2020-07-17T12:09:59.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.414 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:09:59.414 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.414 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:59.414 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.414 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.415 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:59.415 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:59.415 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.415 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:59.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.416 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.416 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:59.416 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:59.416 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.417 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.417 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:09:59.417 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.418 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:59.418 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.418 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.419 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:09:59.419 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:09:59.419 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.419 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:59.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.420 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.420 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.16.9", 2020-07-17T12:09:59.420 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.555197Z" 2020-07-17T12:09:59.421 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.421 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.422 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:09:59.422 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.422 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:59.422 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.422 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.422 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:59.423 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:59.423 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.423 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:59.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.424 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.424 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:59.424 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:59.424 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.424 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.425 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.425 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:09:59.425 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.425 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:09:59.426 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.426 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.426 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:09:59.427 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:09:59.427 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.428 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:09:59.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.429 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.429 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000014.36.5", 2020-07-17T12:09:59.430 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.878502Z" 2020-07-17T12:09:59.431 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.431 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.432 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:09:59.432 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.433 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:59.433 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.433 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.433 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:59.433 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:59.434 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.434 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:59.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.435 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.435 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:59.435 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:59.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.436 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:09:59.436 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.436 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:59.437 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.437 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.437 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:59.438 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:59.438 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.438 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:59.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.439 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.439 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:59.440 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:59.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.441 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.441 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:09:59.442 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.442 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:59.442 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.443 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.443 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:59.443 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:59.444 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.444 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:59.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.445 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.445 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:59.445 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:59.446 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.446 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.447 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:09:59.447 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.449 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:59.449 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.450 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.450 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:59.450 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:59.450 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.451 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:59.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.451 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.451 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:59.451 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:59.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:09:59.452 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:09:59.452 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:09:59.453 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:09:59.453 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:09:59.453 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:09:59.453 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:09:59.453 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:09:59.454 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:09:59.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.454 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:09:59.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:09:59.454 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:09:59.454 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:09:59.455 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:09:59.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:09:59.455 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:09:59.456 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-17 2020-07-17T12:09:59.456 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-17 zones={test-zone2, test-zone1} 2020-07-17T12:09:59.468 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:09:59.475 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:09:59.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:09:59.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:09:59.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:09:59.477 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:09:59.483 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:09:59.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:09:59.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:09:59.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:09:59.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:59.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:59.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:09:59.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:59.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:59.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:09:59.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:00.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:00.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:00.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:00.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:00.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:00.522 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-17 zones={test-zone2, test-zone1} 2020-07-17T12:10:00.523 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-17 zones={test-zone3, test-zone1} 2020-07-17T12:10:00.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:00.536 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:00.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:00.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:00.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:00.538 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:00.543 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:00.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:00.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:00.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:00.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:00.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:00.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:00.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:00.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:00.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:00.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:01.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:01.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:01.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:01.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:01.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:01.538 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-17 zones={test-zone3, test-zone1} 2020-07-17T12:10:01.538 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-17 zones={test-zone3, test-zone2} 2020-07-17T12:10:01.551 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:01.563 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:01.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:01.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:01.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:01.565 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:01.572 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:01.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:01.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:01.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:01.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:01.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:01.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:01.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:01.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:01.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:02.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:02.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:02.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:02.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:02.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:02.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:02.577 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-17 zones={test-zone3, test-zone2} 2020-07-17T12:10:02.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:02.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:02.579 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:02.620 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:02.621 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:02.853 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:02.854 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:02.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.854 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:02.855 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:02.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.856 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:02.856 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.26.5" 2020-07-17T12:10:02.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.857 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:02.857 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:02.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.858 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:02.859 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.9.9" 2020-07-17T12:10:02.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.860 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:02.860 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:02.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.861 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:02.861 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000007.21.9" 2020-07-17T12:10:02.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.862 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:02.862 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:02.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.863 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:02.864 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:02.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.865 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:02.865 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:02.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.867 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:02.867 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:02.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:02.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:02.868 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:02.868 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:02.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:02.869 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:02.869 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:02.870 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:02.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:02.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:02.871 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:02.916 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:02.917 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:03.119 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.120 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:03.121 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.122 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.124 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:03.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.26.5", 2020-07-17T12:10:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:54.067686Z" 2020-07-17T12:10:03.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.128 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:03.129 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.130 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.131 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.132 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:03.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.9.9", 2020-07-17T12:10:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:54.723836Z" 2020-07-17T12:10:03.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.135 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:03.136 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.137 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.139 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:03.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.141 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.141 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000007.21.9", 2020-07-17T12:10:03.141 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:54.349124Z" 2020-07-17T12:10:03.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:03.142 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.146 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:03.147 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:03.148 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.149 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:03.150 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.151 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:03.154 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:03.155 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.157 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:03.158 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:03.163 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:03.164 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:03.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:03.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:03.166 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:03.223 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:03.224 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:03.441 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:03.442 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:03.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.443 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:03.443 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:03.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.444 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:03.444 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.26.5" 2020-07-17T12:10:03.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.445 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:03.445 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:03.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.446 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:03.446 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.9.9" 2020-07-17T12:10:03.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.447 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:03.448 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:03.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.449 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:03.449 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000007.21.9" 2020-07-17T12:10:03.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.450 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:03.450 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:03.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.451 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:03.451 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:03.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.452 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:03.453 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:03.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.454 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:03.454 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:03.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.455 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:03.455 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:03.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.456 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:03.456 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:03.458 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:03.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:03.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:03.460 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:03.549 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:03.549 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:03.755 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:03.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.756 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:03.756 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.756 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:03.757 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.757 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.757 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:03.757 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:03.757 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.758 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:03.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.758 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.758 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:03.759 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.760 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:03.760 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.760 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:03.760 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.761 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.761 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:03.761 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:03.761 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.762 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:03.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.762 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.762 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.26.5", 2020-07-17T12:10:03.762 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:54.067686Z" 2020-07-17T12:10:03.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.763 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:03.764 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.764 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:03.764 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.764 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.764 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:03.764 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:03.765 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.765 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:03.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.765 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.766 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:03.766 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.766 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.766 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.767 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:03.767 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.767 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:03.767 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.767 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.768 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:03.768 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:03.768 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.768 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:03.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.769 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.769 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.9.9", 2020-07-17T12:10:03.769 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:54.713932Z" 2020-07-17T12:10:03.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.770 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.770 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:03.770 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.770 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:03.771 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.771 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.771 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:03.771 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:03.771 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.772 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:03.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.772 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.772 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:03.772 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.773 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.773 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.773 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:03.774 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.774 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:03.774 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.774 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.774 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:03.775 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:03.775 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.775 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:03.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.776 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.776 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000007.21.9", 2020-07-17T12:10:03.776 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:54.349124Z" 2020-07-17T12:10:03.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.777 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:03.777 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.777 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:03.777 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.778 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.778 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:03.778 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:03.778 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.778 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:03.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.779 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.779 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:03.779 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.780 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.780 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:03.780 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.781 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:03.781 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.781 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.781 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:03.781 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:03.781 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.782 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:03.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.782 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.782 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:03.783 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.783 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:03.784 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.784 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:03.784 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.784 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.784 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:03.785 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:03.785 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.785 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:03.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.786 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.786 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:03.786 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.787 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:03.787 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.789 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:03.789 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.789 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.789 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:03.789 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:03.790 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.790 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:03.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.790 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.791 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:03.791 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.791 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.791 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:03.792 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:03.793 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:03.793 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:03.793 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:03.793 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:03.793 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:03.794 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:03.794 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:03.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.794 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:03.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:03.795 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:03.795 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:03.795 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:03.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:03.796 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:03.797 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:03.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:03.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:03.799 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:03.881 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:04.102 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:04.103 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:04.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.103 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:04.104 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:04.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.105 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:04.105 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.26.5" 2020-07-17T12:10:04.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.106 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:04.106 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:04.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.107 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:04.108 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.9.9" 2020-07-17T12:10:04.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.109 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:04.109 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:04.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.110 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:04.110 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000007.21.9" 2020-07-17T12:10:04.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.111 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:04.111 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:04.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.112 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:04.113 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:04.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.115 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:04.115 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:04.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.118 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:04.119 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:04.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:04.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:04.119 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:04.120 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:04.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:04.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:04.121 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:04.137 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:04.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:04.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:04.139 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:04.181 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:04.181 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:04.385 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:04.385 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.386 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.387 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.388 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.389 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.390 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.390 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.390 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.390 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.390 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.391 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.391 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.26.5", 2020-07-17T12:10:04.391 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:54.067686Z" 2020-07-17T12:10:04.391 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.391 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.392 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.393 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:04.394 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.398 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.399 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.9.9", 2020-07-17T12:10:04.401 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:54.143380Z" 2020-07-17T12:10:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.402 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.403 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.406 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:04.407 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.408 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.409 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000007.21.9", 2020-07-17T12:10:04.410 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:54.349124Z" 2020-07-17T12:10:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.413 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.414 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.416 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:04.421 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.422 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.423 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.444 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:04.445 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:04.447 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:04.448 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.449 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.449 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:04.450 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:04.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:04.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:04.452 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:04.537 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:04.538 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:04.747 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:04.747 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:04.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.748 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:04.748 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:04.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.749 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:04.749 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.17.9" 2020-07-17T12:10:04.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.749 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:04.750 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:04.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.750 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:04.750 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000010.26.5" 2020-07-17T12:10:04.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.751 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:04.751 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:04.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.752 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:04.752 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000007.24.9" 2020-07-17T12:10:04.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.752 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:04.753 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:04.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.754 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:04.754 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:04.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.754 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:04.754 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:04.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.755 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:04.755 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:04.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:04.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:04.756 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:04.756 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:04.756 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:04.756 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:04.757 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:04.758 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-18 2020-07-17T12:10:04.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:04.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:04.760 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:04.804 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:04.805 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:05.022 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:05.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.023 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:05.023 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.023 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.024 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.024 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.024 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.024 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.024 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.025 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.025 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.025 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.026 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.026 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:05.027 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.027 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:05.027 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.027 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.027 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:05.028 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:05.028 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.028 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:05.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.029 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.029 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.17.9", 2020-07-17T12:10:05.029 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.952625Z" 2020-07-17T12:10:05.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.030 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:05.030 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.030 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.030 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.031 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.031 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.031 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.031 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.031 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.032 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.032 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.032 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.046 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:05.047 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.047 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:05.047 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.047 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.048 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:05.048 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:05.048 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.049 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:05.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.049 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.050 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.26.5", 2020-07-17T12:10:05.050 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:54.394194Z" 2020-07-17T12:10:05.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.052 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:05.052 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.052 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.053 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.053 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.053 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.062 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.063 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.063 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.063 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.063 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.064 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.065 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:05.065 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.065 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:05.070 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.070 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.071 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:05.071 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:05.071 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.071 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:05.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.072 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.072 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000007.24.9", 2020-07-17T12:10:05.072 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:54.349124Z" 2020-07-17T12:10:05.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.073 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:05.073 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.073 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.074 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.074 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.074 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.074 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.074 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.075 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.075 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.075 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.075 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.076 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:05.077 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.077 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.077 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.077 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.077 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.077 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.078 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.078 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.078 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.079 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.079 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.080 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:05.080 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.080 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.080 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.080 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.080 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.081 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.081 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.081 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.082 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.082 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.082 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.083 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:05.083 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.084 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.084 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.085 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.085 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.085 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.085 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.087 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.088 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.088 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.088 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.089 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:05.089 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.090 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.090 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.090 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.090 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.090 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.091 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.091 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.091 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.091 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.092 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.092 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:05.093 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-18 2020-07-17T12:10:05.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:05.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:05.095 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:05.134 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:05.134 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:05.361 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:05.362 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:05.362 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.362 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:05.362 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:05.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.363 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:05.363 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.17.9" 2020-07-17T12:10:05.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.364 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:05.364 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:05.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.364 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:05.365 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000010.26.5" 2020-07-17T12:10:05.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.365 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:05.365 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:05.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.366 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:05.366 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000007.24.9" 2020-07-17T12:10:05.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.367 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:05.367 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:05.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.367 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:05.368 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:05.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.368 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:05.368 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:05.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.369 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:05.369 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:05.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:05.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:05.370 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:05.370 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:05.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:05.370 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:05.370 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:05.371 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-18 2020-07-17T12:10:05.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:05.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:05.376 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:05.429 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:05.430 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:05.644 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.645 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:05.645 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.645 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.645 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.646 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.646 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.646 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.646 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.646 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.647 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.647 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.647 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.648 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:05.648 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.648 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:05.649 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.649 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.649 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:05.649 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:05.649 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.650 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:05.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.650 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.650 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.17.9", 2020-07-17T12:10:05.651 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.952625Z" 2020-07-17T12:10:05.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.651 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:05.652 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.652 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.652 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.652 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.652 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.653 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.653 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.653 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.654 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.656 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.656 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.656 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.658 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:05.658 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.658 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:05.659 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.659 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.659 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:05.660 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:05.660 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.660 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:05.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.661 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.661 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.26.5", 2020-07-17T12:10:05.661 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:54.389987Z" 2020-07-17T12:10:05.661 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.662 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.662 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:05.663 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.663 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.663 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.663 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.664 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.664 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.664 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.665 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.665 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.665 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.666 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.667 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:05.668 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.668 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:05.668 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.668 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.668 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:05.669 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:05.669 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.669 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:05.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.671 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.672 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000007.24.9", 2020-07-17T12:10:05.672 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:54.349124Z" 2020-07-17T12:10:05.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.673 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:05.673 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.673 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.673 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.673 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.674 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.674 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.674 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.675 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.675 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.675 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.675 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.676 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:05.676 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.677 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.677 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.677 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.677 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.677 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.678 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.678 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.678 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.679 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.679 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.680 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:05.680 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.680 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.680 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.681 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.681 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.681 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.681 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.682 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.682 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.682 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.682 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.683 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:05.683 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.684 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.685 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.685 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.685 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.686 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.686 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.687 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.687 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.687 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.687 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.688 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:05.689 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:05.689 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:05.689 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:05.689 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:05.690 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:05.690 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:05.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:05.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.691 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:05.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.692 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:05.692 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:05.692 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:05.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.693 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:05.693 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-18 2020-07-17T12:10:05.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:05.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:05.695 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:05.774 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:05.775 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:05.988 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:05.988 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:05.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.989 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:05.989 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:05.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.990 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:05.990 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.17.9" 2020-07-17T12:10:05.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.991 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:05.991 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:05.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.991 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:05.992 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.13.9" 2020-07-17T12:10:05.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.992 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:05.992 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:05.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.993 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:05.993 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000015.41.5" 2020-07-17T12:10:05.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.994 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:05.994 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:05.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.994 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:05.995 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:05.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.995 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:05.995 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:05.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.996 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:05.996 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:05.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:05.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:05.997 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:05.997 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:05.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:05.997 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:05.997 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:05.999 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:10:06.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:06.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:06.001 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:06.090 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:06.091 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:06.299 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.17.9", 2020-07-17T12:10:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:53.952625Z" 2020-07-17T12:10:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.13.9", 2020-07-17T12:10:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:54.143380Z" 2020-07-17T12:10:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.321 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:06.322 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.322 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:06.322 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.322 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.323 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:06.323 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:06.323 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000015.41.5", 2020-07-17T12:10:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:08:54.472492Z" 2020-07-17T12:10:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.357 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:06.357 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:10:06.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:06.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:06.359 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:06.400 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:06.401 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:06.620 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.17.9" 2020-07-17T12:10:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.13.9" 2020-07-17T12:10:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000015.41.5" 2020-07-17T12:10:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:06.639 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:06.640 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:10:06.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:06.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:06.643 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:06.694 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:06.695 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:06.914 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:06.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.915 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:06.915 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.916 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:06.916 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.917 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.917 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:06.917 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:06.918 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.918 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:06.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.919 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.919 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:06.919 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.920 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.920 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.921 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:06.922 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.922 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:06.922 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.923 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.923 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:06.923 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:06.924 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.924 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:06.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.925 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.925 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.17.9", 2020-07-17T12:10:06.926 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.952625Z" 2020-07-17T12:10:06.926 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.926 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.927 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:06.928 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.928 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:06.928 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.929 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.929 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:06.929 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:06.932 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.933 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:06.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.934 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.934 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:06.934 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.935 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.935 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.936 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:06.936 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.936 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:06.936 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.937 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.937 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:06.937 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:06.937 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.938 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:06.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.938 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.13.9", 2020-07-17T12:10:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:54.143380Z" 2020-07-17T12:10:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.941 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:06.941 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.941 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:06.942 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.942 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.942 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:06.942 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:06.944 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.945 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:06.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.945 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.946 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:06.946 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.946 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.946 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.947 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.947 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:06.947 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.948 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:06.948 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.948 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.949 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:06.949 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:06.949 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.950 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:06.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.951 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.951 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000015.41.5", 2020-07-17T12:10:06.951 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:54.472492Z" 2020-07-17T12:10:06.952 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.952 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.953 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:06.954 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.954 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:06.955 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.955 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.956 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:06.957 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:06.957 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.958 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:06.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.958 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.959 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:06.959 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.959 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.959 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.960 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.960 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:06.961 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.961 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:06.961 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.961 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.962 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:06.962 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:06.962 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.963 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:06.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.963 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.964 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:06.964 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.964 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.965 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.965 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:06.966 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.966 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:06.966 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.967 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.967 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:06.967 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:06.967 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.968 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:06.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.969 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.969 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:06.969 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.969 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.970 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.970 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:06.971 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.972 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:06.973 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.973 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.973 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:06.974 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:06.974 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.974 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:06.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.975 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.975 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:06.976 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.976 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.976 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:06.977 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:06.977 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:06.977 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:06.978 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:06.978 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:06.978 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:06.979 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:06.979 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:06.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.980 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:06.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:06.980 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:06.980 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:06.980 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:06.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:06.981 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:06.989 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:10:06.990 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-18 zones={test-zone2, test-zone1} 2020-07-17T12:10:07.006 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:07.014 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:07.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:07.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:07.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:07.016 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:07.022 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:07.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:07.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:07.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:07.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:07.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:07.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:07.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:07.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:07.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:07.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:07.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:07.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:07.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:07.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:07.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:08.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-18 zones={test-zone2, test-zone1} 2020-07-17T12:10:08.018 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-18 zones={test-zone3, test-zone1} 2020-07-17T12:10:08.027 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:08.034 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:08.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:08.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:08.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:08.036 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:08.041 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:08.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:08.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:08.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:08.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:08.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:08.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:08.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:08.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:08.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:08.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:08.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:08.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:08.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:08.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:08.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:09.020 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-18 zones={test-zone3, test-zone1} 2020-07-17T12:10:09.021 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-18 zones={test-zone3, test-zone2} 2020-07-17T12:10:09.033 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:09.038 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:09.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:09.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:09.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:09.041 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:09.050 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:09.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:09.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:09.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:09.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:09.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:09.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:09.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:09.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:09.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:09.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:09.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:09.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:09.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:09.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:09.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:10.042 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-18 zones={test-zone3, test-zone2} 2020-07-17T12:10:10.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:10.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:10.692 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:10.732 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:10.937 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000011.31.5" 2020-07-17T12:10:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.14.5" 2020-07-17T12:10:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.26.5" 2020-07-17T12:10:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.945 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:10.947 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:10.948 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:10:10.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:10.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:10.950 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:10.999 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:11.182 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:11.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.183 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:11.183 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.183 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:11.183 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.184 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.184 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:11.184 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:11.184 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.185 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:11.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.185 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.187 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:11.187 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.187 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:11.187 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.187 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.188 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:11.188 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:10:11.188 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000011.31.5", 2020-07-17T12:10:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.101814Z" 2020-07-17T12:10:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:11.191 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.191 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:11.191 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.191 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.191 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:11.192 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:11.192 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.192 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:11.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.192 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.193 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:11.193 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:11.193 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.193 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.194 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:11.194 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.194 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:11.194 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.195 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.195 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:11.195 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:10:11.195 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.196 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:11.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.197 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.197 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.14.5", 2020-07-17T12:10:11.197 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.180239Z" 2020-07-17T12:10:11.197 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:11.201 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:11.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.202 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:11.202 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.202 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:11.202 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.202 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.203 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:11.203 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:10:11.203 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.203 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.26.5", 2020-07-17T12:10:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.249633Z" 2020-07-17T12:10:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:11.206 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.206 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.206 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:11.206 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:11.206 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.207 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:11.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.207 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.208 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:11.208 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:11.208 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.209 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:11.210 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.210 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:11.210 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.211 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.211 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:11.211 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:11.212 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.212 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:11.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.213 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.213 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:11.214 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:11.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.215 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:11.215 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.216 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:11.216 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.216 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.217 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:11.217 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:11.217 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.218 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:11.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.218 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.219 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:11.219 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:11.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.220 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:11.221 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.222 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:11.222 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.223 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.223 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:11.223 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:11.223 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.224 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:11.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.225 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.225 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:11.225 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:11.225 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:11.227 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:11.227 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:11.227 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:11.227 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:11.228 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:11.228 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:11.228 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:11.228 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:11.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.229 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:11.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:11.230 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:11.230 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:11.230 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:11.231 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.231 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.231 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:11.231 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:11.232 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:10:11.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:11.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:11.235 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:11.278 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:11.279 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:11.872 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:11.873 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:11.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.873 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:11.873 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000011.31.5" 2020-07-17T12:10:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.14.5" 2020-07-17T12:10:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.26.5" 2020-07-17T12:10:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.878 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:11.878 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:11.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.879 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:11.879 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:11.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.879 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:11.898 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:11.899 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:10:11.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:11.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:11.901 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:11.985 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:11.986 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:12.188 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:12.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.189 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:12.189 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.189 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:12.190 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.190 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.190 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:12.190 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:12.190 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.191 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:12.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.191 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.191 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:12.191 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.192 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:12.193 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.193 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:12.193 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.193 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.193 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:12.194 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:10:12.194 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.194 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:12.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.195 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.195 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000011.31.5", 2020-07-17T12:10:12.195 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.101814Z" 2020-07-17T12:10:12.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.196 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:12.196 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.197 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:12.197 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.197 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.197 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:12.197 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:12.198 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.198 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:12.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.198 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.198 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:12.199 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.199 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.199 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.200 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:12.200 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.200 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:12.200 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.201 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.201 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:12.201 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:10:12.201 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.14.5", 2020-07-17T12:10:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.180239Z" 2020-07-17T12:10:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:12.205 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.205 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.205 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:12.205 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:12.205 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.206 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:12.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.206 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.206 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:12.206 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.207 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:12.208 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.208 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:12.208 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.208 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.208 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:10:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.26.5", 2020-07-17T12:10:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.249633Z" 2020-07-17T12:10:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.230 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:12.231 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:10:12.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:12.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:12.232 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:12.307 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:12.308 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:12.534 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:12.535 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:12.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.535 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000011.31.5" 2020-07-17T12:10:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.14.5" 2020-07-17T12:10:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000010.26.5" 2020-07-17T12:10:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:12.544 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:12.545 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:10:12.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:12.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:12.547 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:12.590 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:12.591 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:12.802 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:12.802 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.803 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:12.803 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.803 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:12.803 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.803 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.804 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.804 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.804 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.804 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.805 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.805 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:12.805 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.805 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.806 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.806 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:12.806 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.807 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:12.807 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.807 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.807 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.807 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.807 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.808 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.808 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.808 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000011.31.5", 2020-07-17T12:10:12.809 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.101814Z" 2020-07-17T12:10:12.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.810 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:12.810 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.810 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:12.810 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.810 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.810 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.811 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.811 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.811 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.812 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.812 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:12.812 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.812 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.812 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.813 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:12.813 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.813 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:12.813 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.814 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.814 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.814 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.814 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.814 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.815 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.815 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.14.5", 2020-07-17T12:10:12.815 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.180239Z" 2020-07-17T12:10:12.815 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.816 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:12.816 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.817 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:12.817 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.817 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.817 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.817 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.817 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.818 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.818 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.818 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:12.819 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.819 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.819 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.819 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:12.820 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.820 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:12.820 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.820 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.820 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.821 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.821 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.821 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.821 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.822 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000010.26.5", 2020-07-17T12:10:12.822 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.249633Z" 2020-07-17T12:10:12.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.823 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:12.823 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.823 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:12.823 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.823 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.824 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.824 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.824 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.824 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.825 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.825 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:12.825 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.825 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.826 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:12.826 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.826 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:12.827 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.827 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.827 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.827 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.827 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.828 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.828 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.828 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:12.828 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.829 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.829 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.829 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:12.830 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.830 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:12.830 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.830 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.830 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.831 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.831 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.831 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.831 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.832 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:12.832 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.833 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:12.833 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.833 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:12.833 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.833 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.834 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.857 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.857 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.857 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.858 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.858 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:12.858 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.859 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:12.859 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:12.859 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:12.859 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:12.860 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:12.860 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:12.860 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:12.860 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:12.860 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:12.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.861 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:12.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:12.861 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:12.861 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:12.861 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:12.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:12.862 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:12.863 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-16 2020-07-17T12:10:12.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:12.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:12.865 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:12.945 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:12.945 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:13.164 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:13.165 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:13.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.165 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:13.165 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.166 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:13.166 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.18.5" 2020-07-17T12:10:13.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.167 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:13.167 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.167 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.167 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:13.167 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000011.33.5" 2020-07-17T12:10:13.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.168 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:13.168 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.169 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:13.169 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000008.26.5" 2020-07-17T12:10:13.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.169 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:13.170 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.170 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:13.170 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.171 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:13.171 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.172 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:13.172 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.172 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:13.173 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.173 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:13.173 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:13.173 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:13.174 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-16 2020-07-17T12:10:13.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:13.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:13.176 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:13.219 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:13.419 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.420 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.421 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.421 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.421 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.421 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.421 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.422 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.422 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.422 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.423 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.424 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:13.424 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.424 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:13.424 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.424 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.425 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:13.425 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:10:13.425 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.426 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:13.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.426 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.426 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.18.5", 2020-07-17T12:10:13.426 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.101814Z" 2020-07-17T12:10:13.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.427 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.427 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:13.427 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.428 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.428 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.428 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.428 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.428 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.428 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.429 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.429 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.429 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.430 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.430 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.430 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.431 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:13.431 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.431 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:13.431 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.431 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.431 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:13.432 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:10:13.432 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.432 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:13.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.433 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.433 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000011.33.5", 2020-07-17T12:10:13.433 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.180239Z" 2020-07-17T12:10:13.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.434 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:13.434 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.434 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.434 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.435 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.435 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.435 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.435 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.435 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.436 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.436 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.436 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.437 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:13.437 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.437 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:13.438 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.438 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.438 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:13.438 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:10:13.438 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.439 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:13.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.439 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.439 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000008.26.5", 2020-07-17T12:10:13.439 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.249633Z" 2020-07-17T12:10:13.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.440 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:13.441 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.441 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.441 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.441 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.441 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.441 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.442 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.442 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.442 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.443 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.443 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.443 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.443 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.444 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:13.444 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.444 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.444 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.444 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.445 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.445 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.445 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.445 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.446 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.446 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.446 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.447 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:13.447 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.447 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.447 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.448 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.448 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.448 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.448 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.449 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.449 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.449 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.449 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.450 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:13.451 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.451 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.452 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.452 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.452 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.452 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.452 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.453 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.453 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.453 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.454 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.455 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:13.455 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.455 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.455 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.455 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.455 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.456 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.456 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.456 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.457 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.457 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.457 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.459 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.459 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:13.460 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-16 2020-07-17T12:10:13.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:13.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:13.462 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:13.503 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:13.504 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:13.732 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:13.732 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.18.5" 2020-07-17T12:10:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000011.33.5" 2020-07-17T12:10:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000008.26.5" 2020-07-17T12:10:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:13.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:13.741 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:13.741 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:13.741 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-16 2020-07-17T12:10:13.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:13.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:13.743 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:13.788 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:13.788 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:13.985 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:10:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.18.5", 2020-07-17T12:10:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.101814Z" 2020-07-17T12:10:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:10:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000011.33.5", 2020-07-17T12:10:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.180239Z" 2020-07-17T12:10:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:10:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000008.26.5", 2020-07-17T12:10:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.249633Z" 2020-07-17T12:10:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.025 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:14.025 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-16 2020-07-17T12:10:14.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:14.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:14.027 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:14.105 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:14.106 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:14.316 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:14.316 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:14.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.316 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:14.317 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.317 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:14.317 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.14.5" 2020-07-17T12:10:14.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.318 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:14.318 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.319 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:14.320 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000005.19.5" 2020-07-17T12:10:14.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.320 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:14.321 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.322 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:14.322 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000015.41.5" 2020-07-17T12:10:14.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.323 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:14.323 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.324 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:14.324 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.325 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:14.326 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.326 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:14.327 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.328 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:14.328 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.328 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:14.329 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:14.331 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-16 2020-07-17T12:10:14.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:14.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:14.333 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:14.414 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:14.414 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:14.608 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:14.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.609 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:14.609 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.609 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.610 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.610 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.610 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.610 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.610 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.611 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.611 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.611 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.611 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.612 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:14.612 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.613 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:14.613 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.613 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.613 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:14.613 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:10:14.614 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.614 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:14.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.614 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.615 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.14.5", 2020-07-17T12:10:14.615 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.101814Z" 2020-07-17T12:10:14.615 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.615 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.616 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:14.616 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.616 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.616 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.616 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.617 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.617 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.617 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.617 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.618 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.618 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.618 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.619 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:14.619 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.619 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:14.619 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.620 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.620 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:14.620 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:10:14.620 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.621 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:14.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.621 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.621 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000005.19.5", 2020-07-17T12:10:14.621 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.180239Z" 2020-07-17T12:10:14.621 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.622 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:14.622 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.623 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.623 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.623 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.623 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.623 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.623 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.624 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.624 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.624 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.625 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.625 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:14.626 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.626 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:14.626 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.626 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.626 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:14.627 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:10:14.627 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.627 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:14.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.627 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.628 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000015.41.5", 2020-07-17T12:10:14.628 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.249633Z" 2020-07-17T12:10:14.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.629 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:14.629 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.629 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.629 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.629 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.630 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.630 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.630 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.630 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.631 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.631 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.631 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.632 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:14.632 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.632 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.632 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.633 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.633 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.633 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.633 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.634 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.634 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.634 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.634 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.635 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:14.635 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.635 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.636 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.636 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.636 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.636 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.636 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.637 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.637 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.638 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.638 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.639 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:14.639 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.640 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.640 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.640 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.641 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.641 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.641 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.641 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.642 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.642 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.644 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.645 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:14.645 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:14.645 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:14.646 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:14.646 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:14.646 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:14.646 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:14.646 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:14.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.647 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:14.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.647 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:14.647 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:14.647 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:14.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.648 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:14.649 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-16 2020-07-17T12:10:14.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:14.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:14.652 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:14.722 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:14.723 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:14.945 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:14.945 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:14.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.946 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:14.946 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.947 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:14.947 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.14.5" 2020-07-17T12:10:14.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.947 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:14.947 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.948 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:14.948 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000005.19.5" 2020-07-17T12:10:14.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.949 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:14.949 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.950 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:14.950 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000015.41.5" 2020-07-17T12:10:14.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.951 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:14.951 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.952 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:14.952 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.952 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:14.953 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.953 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:14.954 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:14.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:14.955 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:14.956 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:14.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:14.956 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:14.956 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:14.957 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-16 2020-07-17T12:10:14.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:14.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:14.960 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:15.005 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:15.006 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:15.227 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:15.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.228 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:15.228 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.229 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:15.229 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.229 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.230 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:15.230 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:15.230 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.231 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:15.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.232 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.232 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:15.232 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:15.232 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.233 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.233 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.234 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:15.234 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.234 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:15.234 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.235 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.235 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:15.235 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "wa3V2X92bh5.543kEYDmEH01HCP4MDb", 2020-07-17T12:10:15.236 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.236 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:15.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.236 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.237 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.14.5", 2020-07-17T12:10:15.237 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.101814Z" 2020-07-17T12:10:15.237 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.237 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.238 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:15.238 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.238 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:15.238 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.239 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.239 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:15.239 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:15.239 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.240 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:15.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.240 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.240 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:15.240 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:15.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.241 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:15.241 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.242 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:15.242 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.242 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.242 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:15.242 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "L5cXh9EWrl.KnVxuxgEzB9MZYAo1hx.", 2020-07-17T12:10:15.243 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.243 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:15.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.243 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.243 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000005.19.5", 2020-07-17T12:10:15.244 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.180239Z" 2020-07-17T12:10:15.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.245 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:15.245 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.246 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:15.246 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.246 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.247 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:15.247 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:15.247 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.248 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:15.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.248 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.248 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:15.248 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:15.248 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.249 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:15.249 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.250 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:15.250 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.250 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.250 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:15.250 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "v62x6Omd2ec-jEYHhhp3cwzTjNxukrF", 2020-07-17T12:10:15.250 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.251 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:15.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.251 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.251 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000015.41.5", 2020-07-17T12:10:15.252 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.249633Z" 2020-07-17T12:10:15.252 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.252 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.253 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:15.253 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.253 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:15.253 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.254 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.254 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:15.254 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:15.254 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.255 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:15.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.255 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.255 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:15.256 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:15.256 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.256 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.256 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:15.257 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.257 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:15.257 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.257 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.257 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:15.258 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:15.258 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.258 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:15.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.259 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.259 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:15.259 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:15.259 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.259 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.260 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:15.260 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.260 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:15.260 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.261 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.261 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:15.261 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:15.261 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.261 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:15.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.262 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.262 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:15.262 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:15.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.263 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:15.264 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.265 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:15.265 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.265 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.265 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:15.266 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:15.266 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.266 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:15.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.266 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.267 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:15.267 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:15.267 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.267 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:15.268 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:15.268 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:15.268 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:15.268 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:15.268 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:15.269 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:15.269 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:15.269 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:15.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.269 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:15.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:15.270 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:15.270 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:15.270 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:15.270 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.270 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:15.271 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:15.271 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-16 2020-07-17T12:10:15.272 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-16 zones={test-zone2, test-zone1} 2020-07-17T12:10:15.281 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:15.288 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:15.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:15.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:15.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:15.290 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:15.296 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:15.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:15.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:15.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:15.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:15.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:15.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:15.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:15.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:15.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:15.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:15.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:15.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:16.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:16.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:16.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:16.290 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-16 zones={test-zone2, test-zone1} 2020-07-17T12:10:16.291 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-16 zones={test-zone3, test-zone1} 2020-07-17T12:10:16.300 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:16.305 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:16.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:16.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:16.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:16.306 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:16.312 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:16.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:16.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:16.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:16.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:16.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:16.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:16.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:16.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:16.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:16.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:16.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:16.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:17.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:17.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:17.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:17.303 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-16 zones={test-zone3, test-zone1} 2020-07-17T12:10:17.304 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-16 zones={test-zone3, test-zone2} 2020-07-17T12:10:17.316 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:17.322 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:17.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:17.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:17.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:17.324 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:17.331 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:17.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:17.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:17.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:17.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:17.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:17.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:17.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:17.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:17.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:17.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:17.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:17.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:18.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:18.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:18.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:18.305 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-16 zones={test-zone3, test-zone2} 2020-07-17T12:10:18.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:18.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:18.307 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:18.348 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:18.560 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:18.561 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:18.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.561 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:18.561 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:18.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.562 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:18.562 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000011.27.5" 2020-07-17T12:10:18.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.563 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:18.563 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:18.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.563 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:18.564 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.13.5" 2020-07-17T12:10:18.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.564 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:18.564 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:18.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.565 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:18.565 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000008.22.5" 2020-07-17T12:10:18.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.566 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:18.566 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:18.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.566 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:18.567 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:18.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.567 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:18.567 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:18.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.568 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:18.568 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:18.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:18.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:18.569 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:18.569 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:18.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:18.569 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:18.569 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:18.574 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:10:18.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:18.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:18.575 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:18.621 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:18.622 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:18.850 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:18.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.850 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:18.851 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.851 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:18.851 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.851 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.851 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:18.852 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:18.852 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.852 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:18.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.852 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.853 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:18.853 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:18.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.854 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:18.854 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.855 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:18.855 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.855 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.855 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:18.855 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:10:18.855 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.856 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:18.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.856 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.856 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000011.27.5", 2020-07-17T12:10:18.857 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.320103Z" 2020-07-17T12:10:18.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.857 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:18.858 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.858 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:18.858 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.858 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.858 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:18.859 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:18.859 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.859 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:18.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.859 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.860 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:18.860 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:18.860 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.860 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.861 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:18.861 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.861 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:18.861 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.861 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.862 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:18.862 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:10:18.862 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.862 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:18.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.863 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.863 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.13.5", 2020-07-17T12:10:18.863 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.393318Z" 2020-07-17T12:10:18.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.864 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:18.864 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.864 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:18.865 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.865 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.865 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:18.865 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:18.865 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.866 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:18.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.866 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.866 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:18.866 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:18.867 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.867 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.867 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:18.867 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.868 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:18.868 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.868 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.868 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:18.868 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:10:18.869 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.869 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:18.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.869 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.869 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000007.19.9", 2020-07-17T12:10:18.870 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:10:18.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.871 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:18.871 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.871 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:18.871 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.871 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.871 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:18.872 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:18.872 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.872 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:18.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.873 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.873 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:18.873 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:18.873 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.873 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.874 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:18.874 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.874 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:18.874 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.875 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.875 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:18.875 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:18.875 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.875 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:18.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.876 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.876 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:18.876 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:18.876 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.876 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.877 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:18.877 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.877 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:18.878 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.878 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.878 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:18.878 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:18.878 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.879 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:18.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.879 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.879 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:18.879 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:18.880 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.880 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.880 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:18.880 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.881 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:18.882 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.882 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.882 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:18.882 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:18.882 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.883 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:18.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.883 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.883 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:18.884 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:18.884 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.884 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:18.884 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:18.885 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:18.885 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:18.885 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:18.885 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:18.885 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:18.886 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:18.886 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:18.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.886 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:18.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:18.886 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:18.887 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:18.887 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:18.887 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.887 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.887 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:18.888 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:18.889 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000008.22.5 target=00000000007.19.9 2020-07-17T12:10:23.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:23.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:23.892 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:23.942 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:23.943 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:24.127 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:24.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.128 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:24.129 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.129 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:24.129 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.130 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.130 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:24.130 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:24.131 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.131 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:24.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.132 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.132 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:24.133 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:24.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.134 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:24.135 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.135 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:24.135 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.136 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.137 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:24.137 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:10:24.137 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.138 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:24.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.139 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.139 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000011.27.5", 2020-07-17T12:10:24.140 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.320103Z" 2020-07-17T12:10:24.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.141 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:24.142 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.142 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:24.142 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.143 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.143 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:24.143 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:24.144 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.144 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:24.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.145 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.145 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:24.146 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:24.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.148 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:24.149 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.155 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:24.155 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.155 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.155 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:24.155 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:10:24.156 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.156 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:24.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.156 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.157 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.13.5", 2020-07-17T12:10:24.157 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.393318Z" 2020-07-17T12:10:24.157 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.157 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.158 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:24.158 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.158 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:24.158 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.159 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.159 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:24.159 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:24.159 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.160 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:24.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.160 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.160 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:24.160 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:24.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.161 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:24.162 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.162 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:24.162 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.162 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.162 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:24.163 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:10:24.163 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.163 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:24.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.164 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.164 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000007.19.9", 2020-07-17T12:10:24.164 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:08:53.756251Z" 2020-07-17T12:10:24.164 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.164 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.165 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:24.165 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.165 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:24.166 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.166 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.166 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:24.166 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:24.166 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.167 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:24.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.167 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.167 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:24.168 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:24.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.169 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:24.169 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.169 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:24.169 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.169 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.170 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:24.170 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:24.170 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.171 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:24.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.171 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.171 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:24.172 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:24.172 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.172 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.173 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:24.173 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.173 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:24.173 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.173 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.174 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:24.174 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:24.174 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.174 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:24.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.175 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.175 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:24.175 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:24.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.176 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:24.176 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.177 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:24.177 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.178 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.178 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:24.178 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:24.179 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.179 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:24.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.179 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.180 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:24.180 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:24.180 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.180 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:24.181 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:24.181 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:24.181 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:24.181 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:24.181 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:24.182 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:24.182 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:24.182 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:24.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.182 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:24.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:24.183 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:24.183 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:24.183 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:24.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.184 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:24.184 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:24.184 WARNING:rgw_multi.tests:bucket lbjidp-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000008.22.5 target=00000000007.19.9 2020-07-17T12:10:29.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:29.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:29.187 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:29.231 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:29.231 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:29.452 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:29.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.452 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:29.452 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.453 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:29.453 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.453 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.453 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:29.453 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:29.454 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.454 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:29.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.454 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.455 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:29.455 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:29.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.458 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:29.458 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.458 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:29.458 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.458 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.459 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:29.459 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:10:29.459 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.459 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:29.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.460 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.460 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000011.27.5", 2020-07-17T12:10:29.460 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.320103Z" 2020-07-17T12:10:29.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.461 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:29.461 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.462 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:29.462 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.462 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.462 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:29.462 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:29.462 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.463 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:29.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.463 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.463 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:29.464 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:29.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.465 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:29.465 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.465 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:29.465 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.465 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.470 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:29.470 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:10:29.470 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.471 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:29.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.471 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.471 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.13.5", 2020-07-17T12:10:29.471 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.393318Z" 2020-07-17T12:10:29.472 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.472 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.472 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.472 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:29.473 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.473 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:29.473 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.473 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.473 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:29.474 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:29.474 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.474 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:29.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.474 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.475 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:29.475 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:29.475 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.475 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.476 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:29.476 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.476 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:29.476 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.476 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.477 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:29.477 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:10:29.477 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.477 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:29.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.478 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.478 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000008.22.5", 2020-07-17T12:10:29.478 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.463761Z" 2020-07-17T12:10:29.478 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.478 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.479 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:29.479 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.479 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:29.480 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.480 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.480 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:29.480 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:29.480 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.481 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:29.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.481 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.481 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:29.481 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:29.482 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.482 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.482 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:29.483 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.483 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:29.483 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.483 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.483 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:29.483 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:29.484 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.484 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:29.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.484 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.484 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:29.485 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:29.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.486 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:29.486 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.486 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:29.486 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.486 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.487 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:29.487 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:29.487 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.487 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:29.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.488 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.488 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:29.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:29.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.489 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:29.489 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.490 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:29.491 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.491 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.492 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:29.492 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:29.492 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.492 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:29.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.493 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.493 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:29.493 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:29.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:29.497 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:29.498 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:29.498 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:29.498 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:29.498 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:29.498 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:29.499 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:29.499 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:29.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.499 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:29.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:29.499 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:29.500 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:29.500 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:29.500 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.500 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.500 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:29.501 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:29.501 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:10:29.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:29.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:29.503 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:29.544 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:29.545 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:29.758 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000011.27.5" 2020-07-17T12:10:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.13.5" 2020-07-17T12:10:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000008.22.5" 2020-07-17T12:10:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:29.770 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:29.771 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:10:29.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:29.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:29.774 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:30.047 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "LrJ1uJJ9DJvETYRWpfshBS0VWCBnxre", 2020-07-17T12:10:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000011.27.5", 2020-07-17T12:10:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.320103Z" 2020-07-17T12:10:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:10:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.13.5", 2020-07-17T12:10:30.061 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.393318Z" 2020-07-17T12:10:30.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:10:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000008.22.5", 2020-07-17T12:10:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.463761Z" 2020-07-17T12:10:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.085 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:30.085 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.086 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:30.086 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.087 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.087 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:30.087 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:30.087 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.088 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:30.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.088 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.088 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:30.089 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.090 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:30.090 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:30.090 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:30.090 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:30.091 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:30.091 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:30.091 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:30.091 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:30.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.092 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:30.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.092 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:30.092 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:30.092 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.093 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:30.094 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:10:30.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:30.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:30.095 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:30.186 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:30.186 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:30.402 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:30.402 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:30.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.402 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:30.403 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:30.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.403 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:30.403 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000011.27.5" 2020-07-17T12:10:30.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.404 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:30.404 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:30.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.405 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:30.405 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.13.5" 2020-07-17T12:10:30.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.406 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:30.406 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:30.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.407 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:30.407 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000008.22.5" 2020-07-17T12:10:30.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.415 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:30.415 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:30.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.416 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:30.416 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:30.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.417 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:30.417 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:30.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.417 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:30.417 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:30.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:30.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:30.418 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:30.418 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:30.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:30.419 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:30.420 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:30.421 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:10:30.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:30.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:30.425 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:30.472 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:30.473 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:30.683 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:30.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.683 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:30.683 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.684 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:30.684 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.684 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.684 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.684 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.685 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.685 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.686 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.686 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:30.686 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.687 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:30.687 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.688 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:30.688 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.688 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.689 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.689 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.689 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.689 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.690 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.690 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000011.27.5", 2020-07-17T12:10:30.690 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.320103Z" 2020-07-17T12:10:30.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.691 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:30.691 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.692 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:30.692 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.692 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.692 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.692 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.693 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.693 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.693 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.699 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.699 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.699 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.700 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.700 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.700 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.13.5", 2020-07-17T12:10:30.700 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.393318Z" 2020-07-17T12:10:30.701 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.701 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.702 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:30.702 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.702 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:30.702 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.703 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.703 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.703 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.703 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.704 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.704 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.704 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:30.704 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.705 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.705 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:30.706 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.706 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:30.706 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.706 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.706 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.706 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.707 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.707 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.708 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.708 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000008.22.5", 2020-07-17T12:10:30.708 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.463761Z" 2020-07-17T12:10:30.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.710 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:30.711 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.712 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:30.712 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.712 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.713 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.713 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.714 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.714 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.715 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.715 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:30.716 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.716 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.716 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.717 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:30.718 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.718 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:30.718 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.719 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.719 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.719 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.720 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.721 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.721 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.722 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:30.722 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.722 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.723 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.724 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:30.724 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.725 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:30.725 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.726 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.726 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.726 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.727 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.727 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.728 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.728 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:30.729 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.729 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.729 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.731 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:30.731 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.732 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:30.732 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.732 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.733 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.739 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.739 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.740 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.741 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.741 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:30.742 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:30.743 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:30.743 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:30.743 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:30.743 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:30.744 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:30.744 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:30.744 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:30.744 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:30.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.745 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:30.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:30.745 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:30.746 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:30.746 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:30.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:30.747 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:30.750 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-17 2020-07-17T12:10:30.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:30.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:30.752 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:30.794 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:30.795 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:31.020 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:31.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.020 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:31.021 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.021 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.021 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:31.022 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.11.5" 2020-07-17T12:10:31.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.022 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:31.023 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.023 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:31.023 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000011.33.5" 2020-07-17T12:10:31.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.024 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:31.024 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.025 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:31.025 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000010.28.5" 2020-07-17T12:10:31.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.026 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:31.026 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.026 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:31.027 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.027 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:31.027 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.028 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:31.028 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.029 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:31.029 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.029 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:31.030 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:31.030 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:31.031 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:10:31.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:31.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:31.032 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:31.078 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:31.078 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:31.272 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:31.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.272 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:31.272 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.273 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.273 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.273 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.273 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.273 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.274 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.274 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.274 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.275 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.275 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.276 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:31.276 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.276 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:31.277 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.277 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.277 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:31.277 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KKBX3XvnfdY4xOQw7oE4r.tBCwfF.YU", 2020-07-17T12:10:31.278 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.278 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:31.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.279 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.279 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.11.5", 2020-07-17T12:10:31.279 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.320103Z" 2020-07-17T12:10:31.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.280 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:31.280 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.281 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.281 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.281 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.281 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.281 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.282 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.282 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.282 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.283 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.283 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.284 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:31.284 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.284 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:31.285 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.285 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.285 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:31.285 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:10:31.285 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.286 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:31.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.286 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.287 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000011.33.5", 2020-07-17T12:10:31.287 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.393318Z" 2020-07-17T12:10:31.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.288 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:31.288 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.288 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.288 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.289 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.289 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.289 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.289 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.290 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.290 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.290 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.291 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.292 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:31.292 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.292 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:31.292 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.293 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.293 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:31.293 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:10:31.293 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.294 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:31.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.294 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.295 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.28.5", 2020-07-17T12:10:31.295 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.463761Z" 2020-07-17T12:10:31.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.296 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:31.296 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.296 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.297 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.297 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.297 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.297 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.297 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.298 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.298 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.299 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.299 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.300 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:31.300 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.300 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.301 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.301 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.301 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.301 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.301 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.302 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.302 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.302 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.302 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.303 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:31.304 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.304 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.304 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.304 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.305 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.305 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.305 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.306 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.306 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.306 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.306 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.307 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:31.307 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.308 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.309 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.309 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.309 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.309 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.310 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.310 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.310 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.311 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.311 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.312 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:31.312 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.312 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.312 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.313 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.313 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.313 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.313 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.314 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.314 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.314 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.315 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.315 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:31.316 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:10:31.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:31.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:31.318 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:31.361 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:31.362 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:31.594 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.11.5" 2020-07-17T12:10:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000011.33.5" 2020-07-17T12:10:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000010.28.5" 2020-07-17T12:10:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.602 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:31.602 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.603 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:31.603 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:31.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:31.604 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:31.604 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:31.604 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:31.604 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:31.605 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:31.606 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:10:31.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:31.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:31.608 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:31.662 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:31.662 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:31.851 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KKBX3XvnfdY4xOQw7oE4r.tBCwfF.YU", 2020-07-17T12:10:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.11.5", 2020-07-17T12:10:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.320103Z" 2020-07-17T12:10:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:10:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000011.33.5", 2020-07-17T12:10:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.393318Z" 2020-07-17T12:10:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:10:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000010.28.5", 2020-07-17T12:10:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.463761Z" 2020-07-17T12:10:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:31.889 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:31.889 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-17 2020-07-17T12:10:31.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:31.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:31.891 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:31.966 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:31.966 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:32.183 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:32.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.184 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.11.5" 2020-07-17T12:10:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:32.185 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.19.5" 2020-07-17T12:10:32.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000015.39.5" 2020-07-17T12:10:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:32.189 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.191 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.192 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:32.192 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:32.193 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-17 2020-07-17T12:10:32.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:32.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:32.195 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:32.278 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:32.279 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:32.487 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:32.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.488 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:32.488 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.488 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:32.488 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.488 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.489 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:32.489 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:32.489 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.489 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:32.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.490 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.490 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:32.490 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:32.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.491 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.491 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:32.491 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.491 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:32.492 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.492 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.492 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:32.492 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KKBX3XvnfdY4xOQw7oE4r.tBCwfF.YU", 2020-07-17T12:10:32.492 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.493 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:32.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.493 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.493 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.11.5", 2020-07-17T12:10:32.494 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.320103Z" 2020-07-17T12:10:32.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.494 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:32.495 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.495 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:32.495 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.495 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.495 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:32.495 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:32.496 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.496 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:32.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.496 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.497 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:32.497 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:32.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.498 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:32.498 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.498 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:32.498 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.498 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.499 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:32.499 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:10:32.499 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.499 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:32.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.500 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.500 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.19.5", 2020-07-17T12:10:32.500 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.393318Z" 2020-07-17T12:10:32.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.501 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:32.501 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.501 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:32.501 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.502 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.502 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:32.502 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:32.502 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.503 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:32.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.503 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.503 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:32.503 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:32.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.504 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:32.504 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.504 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:32.505 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.505 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.505 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:32.505 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:10:32.505 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.506 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:32.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.506 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.506 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000015.39.5", 2020-07-17T12:10:32.506 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.463761Z" 2020-07-17T12:10:32.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.507 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:32.508 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.508 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:32.508 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.508 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.508 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:32.508 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:32.509 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.509 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:32.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.509 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.510 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:32.510 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:32.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.511 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:32.511 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.511 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:32.511 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.511 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.511 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:32.512 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:32.512 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.512 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:32.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.513 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.513 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:32.513 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:32.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.514 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:32.514 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.514 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:32.514 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.515 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.515 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:32.515 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:32.515 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.515 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:32.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.516 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.516 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:32.516 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:32.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.517 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:32.517 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.518 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:32.519 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.519 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.519 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:32.519 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:32.519 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.520 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:32.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.520 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.520 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:32.520 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:32.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.521 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:32.522 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:32.522 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:32.522 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:32.522 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:32.522 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:32.522 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:32.523 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:32.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.523 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:32.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.523 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:32.523 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:32.524 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:32.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.524 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:32.525 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-17 2020-07-17T12:10:32.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:32.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:32.526 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:32.598 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:32.820 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:32.820 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:32.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.820 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:32.821 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.821 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:32.821 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.11.5" 2020-07-17T12:10:32.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.822 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:32.822 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.823 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:32.823 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.19.5" 2020-07-17T12:10:32.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.823 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:32.824 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.824 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:32.824 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000015.39.5" 2020-07-17T12:10:32.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.825 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:32.825 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.826 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:32.826 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.826 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:32.827 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.827 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:32.827 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:32.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:32.828 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:32.828 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:32.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:32.828 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:32.829 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:32.830 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-17 2020-07-17T12:10:32.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:32.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:32.831 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:32.888 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:32.888 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:33.097 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:33.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.098 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:33.098 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.098 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:33.098 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.099 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.099 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:33.099 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:33.099 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.100 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:33.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.100 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.100 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:33.100 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:33.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.101 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:33.102 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.102 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:33.102 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.102 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.102 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:33.103 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "KKBX3XvnfdY4xOQw7oE4r.tBCwfF.YU", 2020-07-17T12:10:33.103 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.103 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:33.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.103 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.104 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.11.5", 2020-07-17T12:10:33.104 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.320103Z" 2020-07-17T12:10:33.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.105 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:33.105 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.105 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:33.105 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.106 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.106 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:33.106 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:33.106 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.107 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:33.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.107 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.107 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:33.108 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:33.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.108 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:33.109 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.109 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:33.109 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.109 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.109 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:33.110 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "SAgTwwHLQtW7a5cOQhsoFZvzO-co8V0", 2020-07-17T12:10:33.110 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.110 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:33.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.111 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.111 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.19.5", 2020-07-17T12:10:33.111 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.393318Z" 2020-07-17T12:10:33.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:33.112 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.112 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:33.112 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.113 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.113 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:33.113 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:33.113 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.114 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:33.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.114 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.114 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:33.114 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:33.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.115 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:33.116 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.116 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:33.116 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.116 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.116 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:33.117 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jWgm0WKSUbN2OfjpUVJAzdn-ZmcA.bE", 2020-07-17T12:10:33.117 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.117 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:33.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.118 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.118 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000015.39.5", 2020-07-17T12:10:33.118 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.463761Z" 2020-07-17T12:10:33.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.119 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:33.119 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.119 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:33.119 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.120 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.120 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:33.120 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:33.120 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.120 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:33.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.121 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.121 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:33.121 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:33.122 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.122 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.122 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:33.122 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.123 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:33.123 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.123 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.123 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:33.124 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:33.124 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.124 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:33.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.125 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.125 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:33.125 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:33.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.126 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:33.126 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.126 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:33.126 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.127 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.127 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:33.127 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:33.127 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.128 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:33.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.128 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.128 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:33.128 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:33.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.129 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:33.129 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.130 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:33.131 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.131 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.131 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:33.131 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:33.132 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.132 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:33.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.132 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.133 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:33.133 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:33.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:33.134 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:33.134 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:33.134 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:33.134 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:33.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:33.136 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:33.136 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:33.136 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:33.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.137 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.137 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:33.137 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:33.137 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-17 2020-07-17T12:10:33.138 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-17 zones={test-zone2, test-zone1} 2020-07-17T12:10:33.149 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:33.156 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:33.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:33.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:33.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:33.158 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:33.163 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:33.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:33.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:33.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:33.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:33.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:33.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:33.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:33.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:33.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:33.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:33.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:33.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:33.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:33.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:34.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:34.175 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-17 zones={test-zone2, test-zone1} 2020-07-17T12:10:34.176 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-17 zones={test-zone3, test-zone1} 2020-07-17T12:10:34.184 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:34.189 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:34.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:34.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:34.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:34.191 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:34.195 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:34.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:34.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:34.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:34.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:34.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:34.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:34.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:34.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:34.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:34.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:34.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:34.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:34.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:34.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:35.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:35.176 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-17 zones={test-zone3, test-zone1} 2020-07-17T12:10:35.177 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-17 zones={test-zone3, test-zone2} 2020-07-17T12:10:35.187 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:35.192 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:35.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:35.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:35.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:35.194 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:35.201 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:35.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:35.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:35.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:35.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:35.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:35.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:35.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:35.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:35.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:35.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:35.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:35.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:35.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:35.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:36.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:36.177 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-17 zones={test-zone3, test-zone2} 2020-07-17T12:10:36.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:36.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:36.179 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:36.227 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:36.228 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:36.428 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:36.429 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:36.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.429 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:36.429 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:36.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.430 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:36.430 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000011.33.5" 2020-07-17T12:10:36.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.431 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:36.431 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:36.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.431 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:36.432 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.12.5" 2020-07-17T12:10:36.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.432 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:36.432 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:36.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.433 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:36.433 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000008.24.5" 2020-07-17T12:10:36.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.434 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:36.434 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:36.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.434 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:36.435 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:36.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.435 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:36.435 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:36.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.436 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:36.436 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:36.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:36.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:36.436 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:36.437 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:36.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:36.437 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:36.437 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:36.438 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:36.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:36.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:36.441 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:36.494 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:36.494 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:36.708 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:36.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.709 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:36.709 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.709 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:36.709 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.709 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.710 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:36.710 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:36.710 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.710 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:36.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.711 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.711 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:36.711 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:36.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.712 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.712 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:36.712 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.712 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:36.713 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.713 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.713 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:36.713 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:36.713 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.714 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:36.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.714 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.714 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000011.33.5", 2020-07-17T12:10:36.715 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.535021Z" 2020-07-17T12:10:36.715 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.715 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.715 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:36.716 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.716 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:36.716 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.716 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.716 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:36.717 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:36.717 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.717 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:36.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.718 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.718 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:36.718 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:36.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.719 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.720 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:36.722 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.722 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:36.722 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.722 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.723 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:36.723 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:36.723 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.723 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:36.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.724 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.724 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.12.5", 2020-07-17T12:10:36.724 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.612891Z" 2020-07-17T12:10:36.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.725 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:36.725 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.725 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:36.725 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.726 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.726 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:36.726 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:36.726 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.727 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:36.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.727 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.727 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:36.727 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:36.727 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.728 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.728 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:36.728 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.729 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:36.729 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.729 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.729 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:36.729 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:36.730 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.730 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:36.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.730 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.730 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000008.24.5", 2020-07-17T12:10:36.731 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.678707Z" 2020-07-17T12:10:36.731 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.731 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.732 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:36.732 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.732 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:36.732 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.732 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.732 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:36.733 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:36.733 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.733 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:36.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.734 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.734 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:36.734 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:36.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.735 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:36.735 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.735 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:36.735 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.735 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.736 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:36.736 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:36.736 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.736 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:36.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.737 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.737 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:36.737 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:36.737 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.737 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.738 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:36.738 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.738 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:36.739 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.739 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.739 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:36.739 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:36.739 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.740 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:36.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.740 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.740 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:36.740 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:36.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.741 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:36.742 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.743 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:36.743 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.744 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.744 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:36.744 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:36.744 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.745 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:36.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.745 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.745 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:36.745 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:36.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:36.746 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:36.747 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:36.747 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:36.747 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:36.747 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:36.747 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:36.747 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:36.748 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:36.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.748 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:36.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:36.748 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:36.749 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:36.749 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:36.749 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.749 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.749 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:36.749 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:36.750 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:36.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:36.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:36.751 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:36.794 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:36.795 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:37.017 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:37.018 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:37.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.018 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:37.018 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.019 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:37.019 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000011.33.5" 2020-07-17T12:10:37.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.019 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:37.020 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.020 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:37.020 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.12.5" 2020-07-17T12:10:37.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.021 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:37.021 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.029 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:37.030 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000008.24.5" 2020-07-17T12:10:37.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.030 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:37.030 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.031 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:37.031 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.032 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:37.032 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.032 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:37.033 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.033 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:37.033 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.034 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:37.034 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:37.035 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:37.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:37.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:37.036 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:37.123 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:37.123 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:37.309 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:37.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.310 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:37.310 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.311 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:37.311 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.311 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:37.312 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.313 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.316 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.317 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.319 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.320 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000011.33.5", 2020-07-17T12:10:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.535021Z" 2020-07-17T12:10:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.321 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.322 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:37.323 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:37.324 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:37.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:37.326 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:37.328 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.329 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:37.330 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.332 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.5", 2020-07-17T12:10:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.612891Z" 2020-07-17T12:10:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.334 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:37.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:37.337 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000008.24.5", 2020-07-17T12:10:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.678707Z" 2020-07-17T12:10:37.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:37.340 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.341 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:37.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:37.344 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.346 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.347 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.348 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:37.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.351 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:37.352 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.353 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.354 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:37.356 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.357 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.359 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:37.359 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:37.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:37.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:37.361 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:37.436 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:37.436 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:37.648 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:37.649 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000011.33.5" 2020-07-17T12:10:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.651 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.12.5" 2020-07-17T12:10:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000008.24.5" 2020-07-17T12:10:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.654 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:37.658 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:37.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:37.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:37.660 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:37.705 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:37.706 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:37.917 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:37.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.918 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:37.918 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.918 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:37.918 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.919 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.919 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.919 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.919 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.919 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.920 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.920 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:37.920 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.920 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.921 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.921 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:37.921 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.922 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:37.922 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.922 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.922 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.923 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.923 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.923 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.924 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.924 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000011.33.5", 2020-07-17T12:10:37.924 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.535021Z" 2020-07-17T12:10:37.924 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.925 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.925 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:37.925 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.926 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:37.926 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.926 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.926 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.926 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.926 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.927 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.927 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.928 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:37.928 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.928 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.928 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.929 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:37.930 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.930 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:37.931 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.931 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.931 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.932 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.932 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.932 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.933 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.933 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.12.5", 2020-07-17T12:10:37.934 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.612891Z" 2020-07-17T12:10:37.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.935 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:37.936 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.936 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:37.936 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.937 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.937 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.937 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.938 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.938 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.939 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.939 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:37.939 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.941 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:37.941 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.942 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:37.942 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.942 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.942 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.943 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.943 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.944 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.944 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.944 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000008.24.5", 2020-07-17T12:10:37.945 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.678707Z" 2020-07-17T12:10:37.945 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.945 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.946 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:37.946 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.947 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:37.947 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.947 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.947 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.948 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.948 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.949 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.949 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.949 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:37.950 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.950 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.950 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.951 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:37.951 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.952 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:37.952 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.952 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.953 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.953 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.953 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.954 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.954 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.955 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:37.955 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.955 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.956 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.956 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:37.957 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.957 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:37.957 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.958 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.958 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.958 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.958 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.959 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.959 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.960 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:37.960 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.960 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.961 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.961 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.962 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:37.962 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.962 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:37.963 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.963 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.963 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.981 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.981 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.982 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.982 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.983 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:37.983 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:37.984 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:37.984 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:37.984 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:37.984 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:37.984 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:37.985 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:37.985 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:37.985 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:37.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.985 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:37.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:37.986 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:37.986 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:37.986 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:37.987 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.987 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.987 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:37.987 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:37.988 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-18 2020-07-17T12:10:37.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:37.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:37.990 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:38.031 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:38.031 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:38.261 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:38.262 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:38.262 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.262 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:38.262 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.263 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:38.263 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.20.5" 2020-07-17T12:10:38.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.264 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:38.264 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.265 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:38.265 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000011.32.5" 2020-07-17T12:10:38.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.266 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:38.266 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.266 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:38.267 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000008.27.5" 2020-07-17T12:10:38.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.267 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:38.267 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.268 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:38.268 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.269 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:38.269 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.269 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:38.270 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.270 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:38.270 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:38.271 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:38.272 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:38.273 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-18 2020-07-17T12:10:38.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:38.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:38.275 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:38.322 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:38.322 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:38.523 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:38.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.524 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:38.524 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.524 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:38.525 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.525 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.525 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:38.525 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:38.525 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.526 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:38.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.526 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.526 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:38.526 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:38.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.527 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:38.528 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.528 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:38.528 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.528 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.528 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:38.528 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:38.529 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.529 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:38.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.529 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.530 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.20.5", 2020-07-17T12:10:38.530 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.535021Z" 2020-07-17T12:10:38.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.531 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:38.531 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.531 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:38.531 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.531 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.532 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:38.532 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:38.532 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.532 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:38.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.533 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.533 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:38.533 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:38.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.534 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:38.534 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.534 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:38.535 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.535 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.535 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:38.535 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:38.535 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000011.32.5", 2020-07-17T12:10:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.612891Z" 2020-07-17T12:10:38.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.537 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:38.537 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.550 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:38.550 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.550 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.550 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:38.551 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:38.551 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.551 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:38.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.552 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.552 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:38.552 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:38.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.553 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:38.553 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.553 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:38.554 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.555 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.555 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:38.555 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:38.555 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.556 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:38.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.556 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.556 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000008.27.5", 2020-07-17T12:10:38.556 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.678707Z" 2020-07-17T12:10:38.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.557 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:38.557 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.558 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:38.558 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.558 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.558 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:38.558 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:38.559 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.559 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:38.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.560 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.560 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:38.560 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:38.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.561 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:38.561 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.561 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:38.561 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.562 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.562 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:38.562 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:38.562 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.563 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:38.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.563 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.563 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:38.563 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:38.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.565 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:38.565 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.566 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:38.566 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.566 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.566 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:38.567 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:38.567 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.567 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:38.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.568 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.568 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:38.568 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:38.568 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.583 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:38.583 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.585 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:38.585 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.586 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.586 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:38.586 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:38.587 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.587 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:38.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.587 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.588 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:38.588 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:38.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:38.592 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:38.592 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-18 2020-07-17T12:10:38.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:38.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:38.594 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:38.636 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:38.636 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:38.868 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:38.868 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:38.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.868 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:38.869 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.869 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.869 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:38.869 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.20.5" 2020-07-17T12:10:38.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.870 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.870 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:38.870 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.871 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.871 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:38.871 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000011.32.5" 2020-07-17T12:10:38.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.871 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.871 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:38.872 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.873 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:38.873 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000008.27.5" 2020-07-17T12:10:38.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.874 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:38.874 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.875 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:38.875 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.878 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:38.878 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.879 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:38.879 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:38.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:38.880 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:38.880 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:10:38.880 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:38.880 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:38.881 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:38.883 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-18 2020-07-17T12:10:38.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:38.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:38.885 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:38.931 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:38.931 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:39.131 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:39.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.132 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:39.132 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.132 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.132 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.132 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.133 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.133 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.133 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.133 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.134 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.134 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.134 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.135 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.135 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:39.135 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.136 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:39.136 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.136 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.136 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:39.136 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:39.136 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.137 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:39.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.137 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.138 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.20.5", 2020-07-17T12:10:39.139 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.535021Z" 2020-07-17T12:10:39.139 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.139 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.140 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:39.140 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.140 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.141 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.147 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.147 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.147 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.148 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.148 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.148 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.149 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.149 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.150 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:39.150 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.151 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:39.151 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.151 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.151 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:39.151 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:39.151 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.152 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:39.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.152 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.152 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000011.32.5", 2020-07-17T12:10:39.153 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.612891Z" 2020-07-17T12:10:39.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.153 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:39.154 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.154 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.154 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.154 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.154 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.155 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.155 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.155 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.155 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.157 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.157 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.159 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:39.159 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.159 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:39.159 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.159 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.159 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:39.160 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:39.160 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.160 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:39.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.161 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.161 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000008.27.5", 2020-07-17T12:10:39.161 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.678707Z" 2020-07-17T12:10:39.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.162 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:39.163 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.163 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.164 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.164 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.164 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.164 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.164 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.165 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.165 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.165 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.166 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.167 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:39.167 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.167 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.167 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.167 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.167 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.168 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.169 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.170 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.170 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.170 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.170 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.170 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.171 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:39.171 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.172 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.172 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.172 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.172 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.172 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.172 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.173 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.173 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.173 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.177 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.178 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:39.178 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.179 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.179 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.180 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.180 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.180 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.180 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.185 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.185 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.185 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.187 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.187 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:39.188 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-18 2020-07-17T12:10:39.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:39.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:39.193 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:39.264 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:39.265 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:39.460 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:39.460 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:39.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.461 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:39.461 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:39.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.462 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:39.463 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.20.5" 2020-07-17T12:10:39.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.464 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:39.464 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:39.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.465 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:39.465 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.16.5" 2020-07-17T12:10:39.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.466 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:39.466 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:39.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.467 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:39.468 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000016.44.5" 2020-07-17T12:10:39.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.469 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:39.469 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:39.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.470 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:39.470 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:39.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.471 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:39.471 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:39.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.472 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:39.472 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:39.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.472 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:39.473 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:39.473 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.473 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:39.473 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:39.474 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:10:39.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:39.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:39.476 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:39.560 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:39.560 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:39.758 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:39.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.759 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:39.760 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.760 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.760 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.761 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.761 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.762 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.762 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.763 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.763 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.764 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.764 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.766 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:39.766 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.767 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:39.767 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.767 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.768 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:39.768 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:39.768 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.769 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:39.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.770 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.770 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.20.5", 2020-07-17T12:10:39.770 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.535021Z" 2020-07-17T12:10:39.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.772 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:39.772 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.773 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.773 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.774 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.774 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.775 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.775 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.776 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.776 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.777 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.777 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.777 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.779 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:39.779 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.779 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:39.780 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.780 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.780 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:39.786 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:39.787 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.787 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:39.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.788 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.788 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.16.5", 2020-07-17T12:10:39.788 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.612891Z" 2020-07-17T12:10:39.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.789 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:39.789 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.790 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.790 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.790 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.790 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.791 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.791 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.792 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.798 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.798 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.798 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.799 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:39.799 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.800 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:39.800 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.800 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.800 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:39.800 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:39.801 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.801 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:39.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.801 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.802 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000016.44.5", 2020-07-17T12:10:39.802 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.678707Z" 2020-07-17T12:10:39.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.803 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:39.803 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.803 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.803 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.803 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.804 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.804 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.804 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.805 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.805 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.806 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.806 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.807 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:39.807 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.807 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.807 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.807 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.808 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.808 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.808 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.808 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.809 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.809 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.809 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.810 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:39.810 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.810 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.810 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.811 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.811 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.811 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.811 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.812 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.812 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.812 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.812 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.813 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:39.814 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.815 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.815 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.815 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.816 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.816 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.816 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.816 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.817 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.817 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.817 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:39.818 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:39.818 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:39.819 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:39.819 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:39.819 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:39.819 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:39.819 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:39.819 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:39.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.820 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:39.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:39.820 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:39.820 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:39.821 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:39.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:39.821 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:39.822 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:10:39.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:39.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:39.823 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:39.899 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:40.104 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.20.5" 2020-07-17T12:10:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.16.5" 2020-07-17T12:10:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000016.44.5" 2020-07-17T12:10:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:40.121 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:40.122 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:10:40.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:40.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:40.125 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:40.170 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:40.171 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:40.393 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:40.394 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.394 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:40.394 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.394 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:40.394 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.395 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.395 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:40.395 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:40.395 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.396 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:40.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.396 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.396 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:40.396 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:40.397 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.397 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.397 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.397 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:40.397 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.398 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:40.398 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.398 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.398 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone1", 2020-07-17T12:10:40.398 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "ODILETG-f9khTO6MVcZIchlPezzkJj3", 2020-07-17T12:10:40.399 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.399 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:40.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.399 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.400 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.20.5", 2020-07-17T12:10:40.400 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.535021Z" 2020-07-17T12:10:40.400 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.400 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.400 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.401 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:40.401 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.401 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:40.401 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.401 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.402 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:40.402 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:40.402 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.402 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:40.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.403 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.403 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:40.403 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:40.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.405 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:40.405 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.406 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:40.406 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.406 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.406 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj-test-zone2", 2020-07-17T12:10:40.407 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "mskxoGvCkxTMWtr5JrOmOJH1flm4Ahz", 2020-07-17T12:10:40.407 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.407 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:40.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.408 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.408 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.16.5", 2020-07-17T12:10:40.408 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.612891Z" 2020-07-17T12:10:40.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.409 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:40.409 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.410 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:40.410 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.410 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.410 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:40.411 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:40.411 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.411 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:40.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.412 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.412 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:40.412 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:40.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.414 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.414 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:40.414 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.414 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:40.415 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.415 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.415 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:40.415 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "Ow6HGnjAYtS-Q1Bk5rIa7OU6TGChXA8", 2020-07-17T12:10:40.416 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.416 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:40.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.417 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.417 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000016.44.5", 2020-07-17T12:10:40.417 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.678707Z" 2020-07-17T12:10:40.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.419 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:40.419 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.420 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:40.420 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.420 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.421 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:40.421 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:40.421 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.422 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:40.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.422 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.423 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:40.423 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:40.423 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.424 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.425 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:40.425 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.425 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:40.426 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.426 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.426 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:40.426 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:40.426 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.427 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:40.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.427 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.428 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:40.428 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:40.428 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.428 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.429 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:40.429 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.429 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:40.429 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.430 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.430 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:40.430 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:40.430 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.431 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:40.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.431 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.431 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:40.431 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:40.431 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.432 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.432 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:40.432 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.433 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:40.434 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.434 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.434 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:40.434 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:40.434 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.435 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:40.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.435 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.435 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:40.436 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:40.436 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.436 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:40.437 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:40.437 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:40.437 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:40.437 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:40.437 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:40.438 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:40.438 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:40.438 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:40.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.438 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:40.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:40.439 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:40.439 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:40.439 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:40.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:40.440 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:40.440 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-18 2020-07-17T12:10:40.441 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-18 zones={test-zone2, test-zone1} 2020-07-17T12:10:40.453 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:40.460 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:40.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:40.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:40.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:40.462 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:40.469 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:40.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:40.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:40.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:40.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:40.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:40.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:40.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:40.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:40.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:40.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:40.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:41.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:41.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:41.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:41.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:41.489 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-18 zones={test-zone2, test-zone1} 2020-07-17T12:10:41.490 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-18 zones={test-zone3, test-zone1} 2020-07-17T12:10:41.499 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:41.504 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:41.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:41.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:41.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:41.506 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:41.510 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:41.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:41.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:41.512 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:41.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:41.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:41.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:41.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:41.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:41.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:41.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:42.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:42.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:42.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:42.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:42.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:42.513 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-18 zones={test-zone3, test-zone1} 2020-07-17T12:10:42.513 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-18 zones={test-zone3, test-zone2} 2020-07-17T12:10:42.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:10:42.529 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:42.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:42.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:42.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:42.531 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:10:42.538 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:10:42.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-17T12:10:42.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-17T12:10:42.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-17T12:10:42.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:42.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:42.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:10:42.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:42.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:42.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-17T12:10:42.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:43.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:43.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-17T12:10:43.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:43.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:43.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-17T12:10:43.566 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-18 zones={test-zone3, test-zone2} 2020-07-17T12:10:43.567 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-07-17T12:10:43.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:43.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:43.624 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:43.666 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:43.666 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:43.830 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:10:43.859 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987843.602473_95.1", 2020-07-17T12:10:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:43.602473Z" 2020-07-17T12:10:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:10:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:10:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:10:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:10:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:10:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:10:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:43.862 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:43.863 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:10:43.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:43.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:43.865 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:43.906 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:43.906 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:44.110 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:44.111 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:10:44.111 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:10:44.111 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:10:44.111 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:10:44.112 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:10:44.112 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.112 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:44.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:44.113 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:44.113 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:44.113 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:10:44.113 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987843.602473_95.1", 2020-07-17T12:10:44.113 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:10:44.114 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:10:44.114 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:10:44.114 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.602473Z", 2020-07-17T12:10:44.114 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:44.115 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:44.115 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:44.115 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:10:44.115 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:10:44.116 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:10:44.116 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:10:44.116 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:10:44.116 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:10:44.116 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.117 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:44.117 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:44.117 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:44.117 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:10:44.118 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:10:44.118 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:10:44.118 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:10:44.118 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:10:44.118 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:10:44.119 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:44.119 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:44.120 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:44.120 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:10:44.120 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:10:44.120 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:10:44.120 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:10:44.120 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:10:44.130 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:10:44.130 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.131 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.131 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:44.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.131 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:10:44.131 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:10:44.132 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:10:44.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.132 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:44.133 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:10:44.134 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:10:44.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:44.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:44.136 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:44.177 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:44.178 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:44.373 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:44.373 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:10:44.373 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:10:44.373 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:10:44.374 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:10:44.374 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:10:44.374 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:10:44.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:44.374 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:44.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:44.375 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:44.375 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:44.375 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:10:44.375 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987843.602473_95.1", 2020-07-17T12:10:44.376 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:10:44.376 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:10:44.376 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:10:44.376 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.602473Z", 2020-07-17T12:10:44.376 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:10:44.376 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:44.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:44.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:44.377 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:44.377 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:44.377 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:10:44.378 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:10:44.378 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:10:44.378 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:10:44.378 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:10:44.378 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:10:44.379 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:10:44.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:44.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:44.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:44.380 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:44.380 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:44.385 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:10:44.386 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:10:44.386 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:10:44.387 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:10:44.387 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:10:44.387 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:10:44.388 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:10:44.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:44.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:44.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:44.392 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:44.392 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:44.392 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:10:44.392 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:10:44.393 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:10:44.393 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:10:44.393 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:10:44.393 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:10:44.394 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:10:44.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:44.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:44.394 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:44.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:44.395 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:10:44.395 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:10:44.395 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:10:44.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:44.395 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:44.397 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:10:44.397 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:10:44.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:44.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:44.399 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:44.443 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:44.443 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:44.718 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:10:44.718 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:10:44.718 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:10:44.719 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:10:44.719 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:10:44.719 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:10:44.719 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.720 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:10:44.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:44.720 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:44.720 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:44.721 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:10:44.721 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987843.602473_95.1", 2020-07-17T12:10:44.721 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:10:44.721 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:10:44.721 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:10:44.722 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.602473Z", 2020-07-17T12:10:44.722 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.722 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:44.723 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:44.723 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:44.723 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:10:44.723 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:10:44.723 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:10:44.723 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:10:44.724 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:10:44.724 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:10:44.724 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:44.725 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:44.725 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:44.725 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:10:44.725 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:10:44.726 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:10:44.726 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:10:44.726 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:10:44.726 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:10:44.726 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.727 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:44.727 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:44.727 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:44.728 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:10:44.728 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:10:44.729 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:10:44.729 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:10:44.729 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:10:44.729 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:10:44.730 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:10:44.730 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.730 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.730 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:10:44.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:44.731 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:10:44.731 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:10:44.731 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:10:44.731 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:44.731 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:10:44.733 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:10:44.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:44.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:44.783 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:45.030 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:45.031 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:45.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.031 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:45.031 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.032 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:45.032 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.033 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:45.033 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.034 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:45.034 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.035 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:45.035 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.036 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:45.036 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.6.10" 2020-07-17T12:10:45.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.036 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:45.036 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.037 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:45.037 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.038 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:45.038 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.039 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:45.039 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.039 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:45.039 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.040 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.040 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:45.040 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:45.041 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:10:45.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:45.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:45.045 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:45.093 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:45.093 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:45.284 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:10:45.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.285 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:10:45.285 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.285 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.286 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.286 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.286 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.286 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.287 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.287 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.287 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.288 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.288 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.289 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:10:45.289 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.289 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.289 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.289 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.290 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.290 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.290 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.290 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.291 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.291 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.291 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.291 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.292 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.292 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.292 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:10:45.292 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.292 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.293 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.293 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.293 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.293 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.297 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.298 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.298 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.298 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.298 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.299 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.299 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.299 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:10:45.299 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.300 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.300 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.300 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.300 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.300 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.301 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.301 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.301 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.301 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.302 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.302 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.302 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.303 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:10:45.303 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.303 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.303 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.303 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.303 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.304 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.305 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.305 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.306 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.306 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.306 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.307 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:10:45.307 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.307 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:10:45.308 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.308 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.308 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:10:45.308 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:10:45.308 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.309 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:10:45.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.309 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.309 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.6.10", 2020-07-17T12:10:45.309 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.971909Z" 2020-07-17T12:10:45.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.310 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:10:45.311 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.311 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.311 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.311 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.311 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.311 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.312 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.312 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.312 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.313 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.313 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.313 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.313 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.314 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:10:45.314 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.314 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.316 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.316 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.316 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.316 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.317 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.317 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.317 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:10:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.322 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.322 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.322 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:10:45.324 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.324 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.324 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.324 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.324 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.325 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.325 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.326 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.326 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.327 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.327 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:10:45.328 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:10:45.328 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:10:45.328 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:10:45.328 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:10:45.328 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:10:45.329 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:10:45.329 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:10:45.329 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:10:45.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.329 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:10:45.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:10:45.330 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:10:45.330 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:10:45.330 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:10:45.331 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:10:45.331 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:10:45.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:45.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:45.333 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:45.373 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:45.374 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:45.556 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:10:45.557 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:10:45.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.557 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:45.557 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.558 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:45.558 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.559 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:45.559 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.559 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:45.560 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.560 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:45.560 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.561 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:45.561 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000005.12.9" 2020-07-17T12:10:45.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.562 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:45.562 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.563 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:45.563 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.563 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:45.564 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.564 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:45.564 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.565 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:45.565 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:10:45.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.565 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:10:45.566 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:10:45.567 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:10:45.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:45.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:45.568 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:45.648 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:45.649 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:45.838 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:45.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.848 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:45.848 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.849 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.849 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.849 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.849 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.849 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.851 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.852 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.852 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.852 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.853 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.854 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:45.854 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.854 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.855 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.855 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.855 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.855 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.855 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.857 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:45.858 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.858 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.858 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.858 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.858 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.860 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.860 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.860 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.861 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:45.861 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.861 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.861 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.862 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.862 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.862 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.862 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.862 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.863 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.863 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.863 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.864 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:45.865 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.865 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.865 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.865 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.865 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.866 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.866 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.866 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.867 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.867 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.867 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.868 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:45.868 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.868 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:45.868 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.869 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.869 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:45.869 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:10:45.869 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.869 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:45.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.870 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.870 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.6.10", 2020-07-17T12:10:45.870 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.965969Z" 2020-07-17T12:10:45.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.872 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.872 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.872 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.872 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.872 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.873 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.873 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.873 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.873 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.874 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:45.875 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.875 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.875 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.875 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.875 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.876 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.876 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.883 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.883 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.883 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.884 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:45.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.888 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:45.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:45.888 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:45.888 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:45.888 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:45.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:45.889 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:45.890 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.12.9 target=00000000002.6.10 2020-07-17T12:10:50.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:50.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:50.893 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:50.938 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:50.938 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:51.134 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:51.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.134 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:51.134 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.135 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.135 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.135 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.135 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.135 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.136 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.136 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.136 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.137 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.137 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.138 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:51.138 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.138 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.138 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.138 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.139 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.139 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.139 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.139 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.140 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.140 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.140 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.141 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:51.141 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.141 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.142 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.142 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.142 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.142 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.142 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.143 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.143 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.143 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.143 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.144 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:51.145 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.145 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.145 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.145 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.145 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.145 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.146 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.146 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.146 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.147 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.147 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.148 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:51.148 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.149 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.149 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.149 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.149 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.150 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.150 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.150 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.151 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.151 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.151 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.152 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:51.152 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.152 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:51.152 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.153 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.153 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:51.153 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:10:51.153 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.153 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:51.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.154 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.154 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.6.10", 2020-07-17T12:10:51.154 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.965969Z" 2020-07-17T12:10:51.154 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.155 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:51.155 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.155 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.156 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.156 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.156 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.156 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.156 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.157 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.157 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.157 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.157 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.158 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:51.158 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.159 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.159 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.159 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.159 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.159 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.160 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.160 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.160 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.160 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.161 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.162 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:51.162 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.162 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.162 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.162 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.162 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.163 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.163 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.163 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.163 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.164 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.164 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.164 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.164 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.165 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:51.165 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.165 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.165 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.165 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.166 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.166 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.166 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.167 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.168 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.168 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.168 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:51.169 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:51.169 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:51.169 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:51.169 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:51.170 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:51.170 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:51.170 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:51.170 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:51.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.182 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:51.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:51.182 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:51.182 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:51.183 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:51.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:51.183 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:51.184 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.12.9 target=00000000002.6.10 2020-07-17T12:10:56.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:56.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:10:56.187 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:10:56.229 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:10:56.230 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:10:56.422 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:10:56.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.422 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:10:56.422 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.423 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.423 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.423 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.423 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.423 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:10:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:10:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:10:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:10:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:10:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:10:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:10:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:10:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:10:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.6.10", 2020-07-17T12:10:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.965969Z" 2020-07-17T12:10:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:10:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:10:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:10:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:10:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:10:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:10:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:10:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:10:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:10:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:10:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:10:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:10:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:10:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:10:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:10:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:10:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:10:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:10:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:10:56.461 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:10:56.461 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.12.9 target=00000000002.6.10 2020-07-17T12:11:01.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:01.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:01.464 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:01.508 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:01.508 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:01.706 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:11:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:11:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.6.10", 2020-07-17T12:11:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.965969Z" 2020-07-17T12:11:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.736 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.736 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.736 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.737 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:01.737 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.737 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.737 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.738 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.738 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.738 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.738 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.739 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.739 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.739 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.739 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.740 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:01.740 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.741 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.741 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.741 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.741 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.741 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.741 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.742 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.742 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.742 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.743 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.743 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.743 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.743 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:01.744 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.744 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.744 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.744 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.744 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.745 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.745 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.746 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.746 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.747 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.747 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:01.748 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:01.748 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:01.748 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:01.748 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:01.748 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:01.749 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:01.749 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:01.749 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:01.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.749 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:01.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:01.750 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:01.750 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:01.750 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:01.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:01.751 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:11:01.751 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.12.9 target=00000000002.6.10 2020-07-17T12:11:06.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:06.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:06.754 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:06.799 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:06.800 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:06.996 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:11:06.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:06.997 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:06.998 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:06.998 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:06.998 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:06.998 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:06.999 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:06.999 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:06.999 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:06.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:06.999 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.000 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.000 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.000 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.000 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.002 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:07.002 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.002 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:07.003 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.003 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.003 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.007 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:07.008 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.008 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:07.008 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.009 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.009 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:07.011 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:07.011 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.012 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.014 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.014 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.015 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.016 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:07.016 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.017 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:07.017 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.017 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.017 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:07.017 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:07.017 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.018 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.018 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.018 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:11:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000005.12.9", 2020-07-17T12:11:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:11:07.049 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:11:07.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:07.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:07.050 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:07.091 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:07.091 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:07.291 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:11:07.291 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:11:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.293 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:07.293 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.294 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:07.294 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.295 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:07.295 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.295 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:07.296 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.296 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:07.297 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000006.14.9" 2020-07-17T12:11:07.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.297 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:07.298 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.299 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:07.300 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.301 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:07.301 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:07.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:07.302 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:07.302 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:07.302 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:07.303 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:11:07.303 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:11:07.304 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:11:07.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:07.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:07.306 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:07.352 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:07.352 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:07.554 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:11:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.558 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.558 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.562 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.562 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.577 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.578 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.578 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.579 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.579 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.579 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.580 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:07.581 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.581 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.581 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.582 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.582 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.582 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.582 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.583 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.584 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.584 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.584 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.586 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.587 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.588 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.588 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.588 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.589 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.589 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:07.589 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.590 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.590 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.590 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.590 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.590 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.591 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.591 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.591 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.591 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.592 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.593 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:07.593 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.593 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:07.593 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.593 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.593 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:11:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.600 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.600 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.600 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.600 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.600 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.601 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.601 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.601 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.601 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.602 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.602 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.602 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.603 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.604 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.604 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.604 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.604 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.605 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.605 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.605 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.605 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.606 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:07.606 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.606 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.606 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.606 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.606 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.607 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.607 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.607 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.617 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.618 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.618 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.618 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.618 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.619 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.619 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:07.619 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:07.619 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:07.619 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:07.620 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:07.620 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:07.620 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:07.620 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:07.620 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.620 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:07.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.621 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:07.621 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:07.621 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:07.621 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.622 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.622 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:11:07.623 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:11:07.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:07.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:07.625 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:07.703 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:07.704 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:07.927 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:11:07.927 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:11:07.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:07.929 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.929 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:07.929 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.930 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:07.930 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.931 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:07.931 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.931 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:07.932 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.12.9" 2020-07-17T12:11:07.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.932 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.932 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:07.932 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.933 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:07.933 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.934 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:07.934 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.934 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:07.934 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:07.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:07.935 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:07.935 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:07.935 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:07.936 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:11:07.936 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:11:07.937 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-19 2020-07-17T12:11:07.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:07.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:07.939 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:07.984 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:07.984 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:08.184 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:11:08.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.185 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:08.185 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.185 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.186 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.186 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.186 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.187 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.187 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.188 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.191 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.191 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.191 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.193 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:08.193 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.194 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.194 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.194 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.195 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.195 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.195 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.196 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.197 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.197 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.197 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.199 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:08.199 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.200 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.200 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.200 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.201 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.201 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.201 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.205 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.206 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.206 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.207 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.208 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:08.209 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.209 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.209 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.210 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.210 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.210 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.211 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.211 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.212 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.212 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.212 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.213 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.213 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.214 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:08.214 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.215 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.216 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.216 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.216 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.217 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.217 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.218 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.218 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.218 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.219 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.219 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.219 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.220 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:08.221 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.221 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:08.221 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.222 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.222 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:11:08.222 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:08.223 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.223 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:08.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.224 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.224 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:11:08.225 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:08.225 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.225 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.226 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:08.227 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.227 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.227 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.227 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.227 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.228 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.228 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.228 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.229 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.229 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.229 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.230 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:08.230 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.231 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.231 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.231 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.231 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.231 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.232 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.232 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.233 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.233 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.233 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.233 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.233 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.234 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:08.234 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.234 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.234 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.235 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.235 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.235 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.235 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.242 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.243 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.243 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.244 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.246 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:08.247 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.247 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.247 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.247 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.247 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.255 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:11:08.255 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-19 2020-07-17T12:11:08.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:08.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:08.257 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:08.298 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:08.299 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:08.523 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:11:08.524 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:11:08.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.524 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:08.524 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.525 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:08.525 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.526 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:08.526 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.527 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:08.527 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.528 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:08.528 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.529 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:08.529 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.12.9" 2020-07-17T12:11:08.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.530 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:08.530 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.530 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:08.531 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.531 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:08.531 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.532 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:08.532 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:08.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:08.533 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:08.533 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:11:08.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:08.533 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:11:08.533 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:11:08.542 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-19 2020-07-17T12:11:08.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:08.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:08.543 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:08.588 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:08.786 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:11:08.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.787 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:08.787 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.787 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.787 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.788 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.788 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.788 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.788 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.789 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.789 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.789 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.789 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.790 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.790 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.790 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:08.790 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.791 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.791 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.791 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.791 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.791 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.792 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.792 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.792 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.793 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.793 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.794 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:08.794 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.794 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.795 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.795 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.795 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.795 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.796 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.796 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.799 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.799 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.799 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.800 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:08.800 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.800 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.800 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.801 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.801 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.801 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.801 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.802 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.802 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.802 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.802 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.803 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:08.803 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.804 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.804 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.804 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.804 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.804 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.804 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.805 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.805 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.805 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.806 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.807 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:08.807 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.807 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:08.808 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.808 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.808 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:11:08.808 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:08.809 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.809 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:08.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.810 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.810 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:11:08.810 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:08.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.811 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:08.811 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.811 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.812 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.812 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.812 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.812 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.812 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.813 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.813 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.813 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.813 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.814 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:08.814 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.815 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.815 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.815 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.815 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.815 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.816 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.816 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.816 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.816 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.817 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.819 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:08.820 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.820 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.820 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.820 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.820 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.820 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.821 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.821 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.821 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.822 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.822 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.823 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:08.823 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.823 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.823 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.823 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.824 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.824 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.824 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.826 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.826 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.826 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.827 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:08.828 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:08.828 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:08.828 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:08.829 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:08.829 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:08.830 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:08.831 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:08.831 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:08.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.832 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:08.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:08.832 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:08.833 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:08.833 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:08.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:08.834 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:11:08.835 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-19 2020-07-17T12:11:08.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:08.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:08.837 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:08.913 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:08.913 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:09.113 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:11:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:11:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:09.116 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.118 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.15.9" 2020-07-17T12:11:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:09.119 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.121 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:09.122 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:11:09.123 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:11:09.124 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-19 2020-07-17T12:11:09.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:09.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:09.126 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:09.211 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:09.212 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:09.382 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:11:09.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.383 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:09.383 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.384 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.384 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.384 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.384 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.384 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.384 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.385 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.385 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.385 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.386 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.387 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:09.387 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.387 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.387 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.387 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.388 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.388 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.388 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.388 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.389 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.389 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.389 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.389 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.389 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.391 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:09.391 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.391 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.391 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.392 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.392 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.392 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.392 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.393 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.393 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.393 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.393 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.394 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:09.394 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.395 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.395 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.395 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.395 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.395 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.396 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.396 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.396 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.396 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.397 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.397 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.397 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.399 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:09.400 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.400 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.400 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.400 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.400 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.401 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.401 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.401 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.401 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.402 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.402 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.402 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.402 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.403 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:09.403 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.403 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:09.403 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.403 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.404 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:11:09.404 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:09.404 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.404 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:09.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.405 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.405 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.13.9", 2020-07-17T12:11:09.405 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:09.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.407 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:09.407 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.407 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.407 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.407 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.408 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.408 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.408 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.409 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.410 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.410 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.410 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.411 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:09.411 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.411 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.411 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.412 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.412 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.412 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.412 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.413 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.413 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.413 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.413 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.414 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:09.414 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.415 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.415 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.415 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.415 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.415 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.415 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.416 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.416 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.416 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.416 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.417 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:09.418 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.418 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.418 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.418 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.419 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.419 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.419 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.421 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.422 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.422 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.422 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:09.423 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:09.423 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:09.423 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:09.424 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:09.424 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:09.424 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:09.424 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:09.424 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:09.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.425 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:09.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:09.425 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:09.425 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:09.425 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:09.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:09.426 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:11:09.426 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.15.9 target=00000000003.13.9 2020-07-17T12:11:14.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:14.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:14.429 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:14.475 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:14.475 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:14.667 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:11:14.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.668 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:14.668 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.668 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.669 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.669 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.669 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.669 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.669 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.670 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.670 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.670 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.670 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.671 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:14.672 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.672 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.672 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.672 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.672 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.672 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.673 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.673 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.673 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.679 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.681 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.682 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.683 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.684 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.699 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.700 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.701 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:11:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.705 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.13.9", 2020-07-17T12:11:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:14.732 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:11:14.733 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.15.9 target=00000000003.13.9 2020-07-17T12:11:19.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:19.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:19.736 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:19.780 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:19.780 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:19.984 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:11:19.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:19.985 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:19.985 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:19.985 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:19.986 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:19.986 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:19.986 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:19.986 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:19.986 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:19.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.987 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:19.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.987 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:19.987 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:19.988 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:19.988 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:19.988 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:19.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:19.988 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:19.989 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:19.989 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:19.989 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:19.989 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:19.989 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:19.990 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:19.990 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:19.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.990 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:19.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.991 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:19.991 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:19.991 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:19.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:19.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:19.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:19.992 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:19.992 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:19.992 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:19.992 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:19.992 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:19.993 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:19.993 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:19.993 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:19.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.993 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:19.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.994 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:19.994 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:19.994 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:19.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:19.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:19.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:19.995 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:19.995 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:19.995 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:19.996 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:19.996 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:19.996 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:19.996 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:19.996 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:19.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.997 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:19.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.997 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:19.997 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:19.997 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:19.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:19.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:19.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:19.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:19.998 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:19.998 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:19.999 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:19.999 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:19.999 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:19.999 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:19.999 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:20.000 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:20.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.000 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:20.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.000 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:20.000 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:20.001 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:20.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:20.002 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:20.002 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:20.002 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:20.002 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:20.002 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:20.002 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:11:20.003 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:20.003 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:20.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.003 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:20.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.004 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:20.004 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.13.9", 2020-07-17T12:11:20.004 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:20.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:20.005 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:20.005 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:20.005 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:20.005 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:20.006 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:20.006 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:20.006 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:20.006 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:20.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.013 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:20.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.014 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:20.014 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:20.014 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:20.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:20.015 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:20.015 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:20.016 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:20.016 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:20.016 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:20.017 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:20.017 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:20.017 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:20.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.018 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:20.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.018 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:20.020 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:20.021 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:20.021 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:20.022 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:20.022 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:20.022 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:20.026 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:20.026 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:20.026 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:20.026 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:20.026 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:20.027 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:20.027 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:20.027 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:20.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.028 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:20.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.029 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:20.029 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:20.029 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:20.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:20.030 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:20.030 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:20.030 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:20.031 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:20.031 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:20.031 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:20.032 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:20.032 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:20.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.032 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:20.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:20.033 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:20.033 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:20.033 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:20.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:20.034 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:11:20.034 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.15.9 target=00000000003.13.9 2020-07-17T12:11:25.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:25.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:25.037 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:25.082 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:25.083 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:25.262 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:11:25.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.263 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:25.264 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.264 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.264 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.264 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.264 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.265 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.265 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.265 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.266 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.266 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.266 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:11:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.15.9", 2020-07-17T12:11:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.298 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.298 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.298 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.300 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.300 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.305 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:25.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.305 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:25.305 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.307 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:11:25.307 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-19 2020-07-17T12:11:25.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:25.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:25.310 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:25.384 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:25.385 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:25.585 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:11:25.585 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:11:25.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.586 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:25.586 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.587 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:25.587 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.587 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:25.588 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.588 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:25.588 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.589 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:25.589 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.590 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:25.590 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.15.9" 2020-07-17T12:11:25.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.591 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:25.591 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.591 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:25.591 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.592 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:25.592 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.593 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:25.593 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:25.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:25.593 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:25.594 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:25.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:25.594 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:11:25.594 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:11:25.595 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-19 2020-07-17T12:11:25.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:25.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:25.597 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:25.642 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:25.642 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:25.847 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:11:25.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.848 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:25.848 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.848 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.848 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.849 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.849 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.849 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.849 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.850 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.850 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.850 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.850 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.850 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.851 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:25.851 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.853 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.853 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.855 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:25.855 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.855 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.857 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.857 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.859 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:25.859 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.859 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.859 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.859 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.861 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.861 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.861 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.861 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.862 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:25.862 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.865 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.865 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.865 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.866 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:25.866 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.866 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:25.866 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.866 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.867 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:11:25.867 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:25.867 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.867 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:25.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.868 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.868 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.15.9", 2020-07-17T12:11:25.868 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:25.868 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.868 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.869 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.869 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:25.869 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.869 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.869 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.870 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.870 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.870 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.870 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.871 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.871 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.871 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.871 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.871 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.872 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.872 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:25.872 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.873 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.873 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.873 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.873 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.873 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.874 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.874 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.874 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.875 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.875 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.875 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.875 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.876 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:25.876 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.876 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.876 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.876 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.876 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.877 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.877 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.877 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.878 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.878 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.878 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.878 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.878 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.879 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:25.879 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.879 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.879 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.880 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.880 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.880 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.880 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.882 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.882 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.882 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.883 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.883 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.883 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:25.884 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:25.884 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:25.884 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:25.884 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:25.884 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:25.884 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:25.885 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:25.885 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:25.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.885 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:25.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:25.886 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:25.886 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:25.886 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:25.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:25.887 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:11:25.887 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-19 2020-07-17T12:11:25.888 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-19 zones={test-zone2, test-zone1} 2020-07-17T12:11:25.903 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:11:25.951 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:11:25.952 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:11:25.959 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:11:26.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:11:26.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:11:26.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:11:26.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-19 zones={test-zone2, test-zone1} 2020-07-17T12:11:26.172 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-19 zones={test-zone3, test-zone1} 2020-07-17T12:11:26.180 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:11:26.227 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:11:26.227 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:11:26.234 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:11:26.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:11:26.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:11:26.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:11:26.446 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-19 zones={test-zone3, test-zone1} 2020-07-17T12:11:26.447 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-19 zones={test-zone3, test-zone2} 2020-07-17T12:11:26.460 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:11:26.510 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:11:26.511 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:11:26.519 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:11:26.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:11:26.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:11:26.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:11:26.717 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-19 zones={test-zone3, test-zone2} 2020-07-17T12:11:28.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:28.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:28.206 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:28.247 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:28.248 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:28.456 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:11:28.457 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:11:28.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.457 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:28.457 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:28.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.458 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:28.458 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:28.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.459 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:28.459 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:28.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.459 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:28.460 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:28.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.460 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:28.460 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:28.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.461 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:28.461 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000006.14.9" 2020-07-17T12:11:28.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.462 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:28.462 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:28.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.463 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:28.466 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:28.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.467 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:28.467 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000027.76.11" 2020-07-17T12:11:28.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.467 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:28.467 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:28.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:28.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:28.475 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:28.476 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:28.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:28.476 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:11:28.476 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:11:28.477 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:11:28.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:28.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:28.479 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:28.524 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:28.526 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:28.732 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:11:28.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.732 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:28.733 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.733 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:28.733 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.733 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.733 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:28.734 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:28.734 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.734 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:28.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.734 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.735 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:28.735 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:28.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.736 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:28.736 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.736 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:28.736 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.736 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.737 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:28.737 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:28.737 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.737 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:28.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.738 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.738 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:28.738 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:28.738 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.739 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:28.739 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.739 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:28.740 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.740 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.740 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:28.740 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:28.740 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.741 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:28.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.741 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.741 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:28.741 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:28.743 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.744 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:28.744 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.745 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:28.745 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.745 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.745 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:28.745 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:28.746 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.746 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:28.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.747 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.747 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:28.747 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:28.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.748 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:28.749 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.749 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:28.749 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.749 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.749 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:28.749 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:28.750 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.750 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:28.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.750 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.751 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:28.751 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:28.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.752 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:28.752 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.752 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:28.752 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.752 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.753 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:11:28.753 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:28.753 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.753 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:28.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.754 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.754 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:11:28.754 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:28.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.758 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.758 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:28.758 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.759 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:28.759 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.759 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.759 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:28.759 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:28.759 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.760 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:28.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.760 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.760 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:28.761 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:28.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.761 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:28.762 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.762 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:28.762 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.762 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.762 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:28.763 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:28.763 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.763 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:28.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.763 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.765 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:11:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "MQ.yErR1ERWRUwZuP4MJi-IhHhGeLKQ", 2020-07-17T12:11:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.22.11", 2020-07-17T12:11:28.767 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:26.862710Z" 2020-07-17T12:11:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:28.768 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.769 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:28.776 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:28.777 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:28.778 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:11:28.779 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000027.76.11 target=00000000004.22.11 2020-07-17T12:11:33.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:33.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:33.782 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:33.826 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:33.827 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:34.027 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:11:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.028 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:34.029 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.031 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:34.033 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.034 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.035 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:34.036 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:34.037 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.037 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.038 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.039 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:34.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.041 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.042 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:34.043 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.044 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:34.046 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:34.048 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.049 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.050 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:11:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:34.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:11:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:34.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:34.053 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:34.054 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.055 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.057 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.058 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:34.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.061 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:34.063 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:11:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "MQ.yErR1ERWRUwZuP4MJi-IhHhGeLKQ", 2020-07-17T12:11:34.064 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.065 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.22.11", 2020-07-17T12:11:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:26.862710Z" 2020-07-17T12:11:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.067 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.070 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:34.072 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:34.076 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:34.077 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:34.078 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:34.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:34.080 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:34.081 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:11:34.082 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000027.76.11 target=00000000004.22.11 2020-07-17T12:11:39.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:39.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:39.085 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:39.342 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:11:39.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.343 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:39.343 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.343 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:39.343 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.343 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.344 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:39.344 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:39.344 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.344 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:39.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.345 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.345 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:39.345 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:39.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.346 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:39.347 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.347 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:39.347 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.347 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.347 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:39.347 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:39.348 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.348 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:39.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.348 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.349 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:39.349 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:39.349 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.349 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.350 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.350 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:39.350 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.350 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:39.350 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.350 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.351 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:39.351 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:39.351 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.351 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:39.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.352 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.352 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:39.352 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:39.353 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.353 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.353 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.353 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:39.354 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.354 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:39.354 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.355 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.355 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:39.355 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:39.355 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.356 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.358 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.359 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:11:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.363 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:39.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.363 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.363 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:11:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.365 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:39.365 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.365 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:39.365 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.366 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.366 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:39.366 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:39.369 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.369 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:39.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.370 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.370 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:39.370 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:39.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.371 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:39.372 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.372 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:39.373 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.373 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.373 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:39.374 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:39.374 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.375 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:39.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.375 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.376 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:39.376 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:39.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.377 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.377 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:39.377 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.377 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:39.378 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.378 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.378 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:11:39.378 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "MQ.yErR1ERWRUwZuP4MJi-IhHhGeLKQ", 2020-07-17T12:11:39.378 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.379 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:39.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.379 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.379 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.22.11", 2020-07-17T12:11:39.380 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:26.862710Z" 2020-07-17T12:11:39.380 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.380 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.381 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:39.381 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.381 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:39.381 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.381 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.382 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:39.382 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:39.383 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.384 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:39.385 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.385 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.385 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:39.385 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:39.386 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.386 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.386 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:39.386 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:39.386 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:39.387 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:39.387 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:39.387 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:39.387 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:39.387 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:39.388 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:39.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.388 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:39.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:39.388 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:39.388 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:39.389 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:39.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:39.389 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:11:39.390 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000027.76.11 target=00000000004.22.11 2020-07-17T12:11:44.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:44.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:44.392 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:44.437 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:44.438 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:44.654 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:11:44.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:44.655 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:44.655 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:44.656 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:44.656 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:44.656 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:44.657 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:44.657 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:44.659 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:44.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:44.660 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:44.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:44.660 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:44.661 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:44.661 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:44.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:44.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:44.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:44.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:44.663 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:44.663 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:44.664 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:44.664 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:44.664 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:44.665 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:44.665 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:44.666 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:44.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:44.667 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:44.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:44.668 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:44.669 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:44.669 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:44.670 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:44.670 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:44.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:44.671 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:44.671 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:44.672 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:44.672 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:44.672 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:44.674 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:44.674 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:44.675 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:44.675 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:44.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.003 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:45.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.004 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:45.004 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:45.004 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:45.005 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.005 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:45.006 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:45.006 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:45.007 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:45.007 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:45.007 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:45.008 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:45.008 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:45.008 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:45.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.009 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:45.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.010 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:45.010 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:45.010 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:45.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:45.011 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:45.012 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:45.012 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:45.012 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:45.012 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:45.012 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:45.012 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:45.013 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:45.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.013 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:45.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.013 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:45.014 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:45.014 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:45.014 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.014 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.015 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:45.015 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:45.015 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:45.015 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:45.016 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:11:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.018 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:45.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.019 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:45.019 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:11:45.019 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:45.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.020 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:45.020 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:45.020 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:45.021 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:45.021 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:45.021 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:45.021 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:45.021 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:45.022 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:45.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.023 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:45.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.023 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:45.023 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:45.024 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:45.024 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.024 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:45.029 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:45.029 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:45.029 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:45.029 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:11:45.030 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "MQ.yErR1ERWRUwZuP4MJi-IhHhGeLKQ", 2020-07-17T12:11:45.030 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:45.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.031 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:45.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.031 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:45.032 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.22.11", 2020-07-17T12:11:45.032 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:26.862710Z" 2020-07-17T12:11:45.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:45.033 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:45.034 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:45.034 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:45.034 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:45.035 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:45.036 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:45.036 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:45.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:45.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.037 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:45.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.038 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:45.038 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:45.039 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:45.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:45.040 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:45.041 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:45.041 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:45.041 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:45.042 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:45.042 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:45.042 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:45.043 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:45.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.043 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:45.046 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:11:45.047 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000027.76.11 target=00000000004.22.11 2020-07-17T12:11:50.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:50.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:50.049 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:50.099 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:50.099 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:50.299 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:11:50.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.300 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:50.300 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.300 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:50.301 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.301 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.301 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:50.302 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:50.302 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.303 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:50.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.303 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.304 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:50.304 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.304 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.305 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.306 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:50.306 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.306 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:50.306 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.307 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.307 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:50.307 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:50.308 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.308 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:50.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.309 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.309 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:50.322 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.324 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:50.326 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.326 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:50.327 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.327 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.327 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:50.328 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:50.328 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.329 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:50.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.330 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.330 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:50.330 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.332 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:50.332 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.332 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:50.332 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.332 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.333 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:50.333 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:50.333 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.333 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:50.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.339 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.339 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:50.339 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.340 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.340 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.341 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:50.341 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.341 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:50.341 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.341 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.342 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:50.342 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:50.342 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.342 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:50.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.343 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.343 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:50.343 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.343 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.343 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.344 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.344 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:50.344 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.344 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:50.344 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.345 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.345 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:11:50.345 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:50.345 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.346 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:50.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.346 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.346 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:11:50.347 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:50.347 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.347 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.347 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.347 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:50.348 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.348 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:50.348 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.348 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.348 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:50.349 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:50.349 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.349 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:50.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.354 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:50.354 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.354 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:11:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "MQ.yErR1ERWRUwZuP4MJi-IhHhGeLKQ", 2020-07-17T12:11:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:11:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000045.124.9", 2020-07-17T12:11:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:11:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.358 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:50.358 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.358 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.359 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:50.359 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:50.359 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.360 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:50.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.360 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.360 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:50.361 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.362 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:50.362 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:50.362 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:50.362 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:50.362 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:50.362 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:50.363 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:50.363 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:50.363 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:50.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.363 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:50.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:50.364 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:50.364 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:50.364 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.365 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:50.365 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:11:50.365 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:11:50.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:50.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:50.367 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:50.408 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:50.409 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:50.619 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:11:50.619 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:11:50.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.620 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:50.620 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:50.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.621 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:50.621 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:50.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.622 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:50.622 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:50.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:50.623 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:50.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.624 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:50.625 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000006.14.9" 2020-07-17T12:11:50.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.626 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:50.626 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:50.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.627 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:50.627 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:50.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.628 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:50.628 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000046.126.9" 2020-07-17T12:11:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:50.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.630 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:50.630 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:50.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.631 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:11:50.631 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:11:50.632 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:11:50.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:50.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:50.634 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:50.730 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:50.925 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:11:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:50.925 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.926 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:50.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.929 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.930 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:50.931 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.932 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:50.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.932 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.932 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:50.933 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.934 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:50.934 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.934 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:50.934 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.934 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.935 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:50.935 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:50.935 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.936 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:50.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.936 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.936 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:50.937 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.938 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:50.939 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.939 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:50.939 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.940 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.940 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:50.941 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:50.941 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.942 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:50.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.943 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.943 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:50.944 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.946 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:50.946 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.946 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:50.947 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.947 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.947 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:50.948 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:50.948 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.948 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:50.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.949 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.949 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:50.950 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.951 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:50.951 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.952 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:50.952 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.952 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.952 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:11:50.953 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:11:50.953 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000006.14.9", 2020-07-17T12:11:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.956 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:50.957 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.957 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:50.957 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.958 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.958 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:50.958 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:50.959 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.959 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:50.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.960 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.960 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:50.960 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.962 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:50.962 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.963 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.966 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:50.966 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.966 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:11:50.966 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:11:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "MQ.yErR1ERWRUwZuP4MJi-IhHhGeLKQ", 2020-07-17T12:11:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.967 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:11:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000046.126.9", 2020-07-17T12:11:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.105198Z" 2020-07-17T12:11:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.969 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:50.970 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.970 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.972 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:50.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.972 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.972 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:50.972 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:50.973 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:50.974 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:11:50.974 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:11:50.974 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:11:50.974 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:11:50.974 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:11:50.974 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:11:50.975 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:11:50.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.975 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:11:50.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:50.975 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:11:50.976 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:11:50.976 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:11:50.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:50.977 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:11:50.977 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:11:50.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:50.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:50.979 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:51.273 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:11:51.274 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:11:51.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.274 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:11:51.274 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:51.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.275 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:11:51.275 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:51.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.276 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:11:51.276 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:51.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.277 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:11:51.277 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:51.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.277 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:11:51.280 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:51.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.281 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:11:51.281 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000006.14.9" 2020-07-17T12:11:51.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.282 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:11:51.282 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:51.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.293 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:11:51.293 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:51.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.294 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:11:51.295 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000046.126.9" 2020-07-17T12:11:51.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.295 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:11:51.296 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:51.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:11:51.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:11:51.297 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:11:51.297 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:11:51.297 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:11:51.298 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:11:51.298 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:11:51.298 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:11:51.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:51.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:51.301 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:51.347 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:51.347 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:51.553 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:11:51.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.554 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:51.554 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.554 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:51.555 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.555 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.555 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.555 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.555 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.556 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.556 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.557 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:51.557 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:51.557 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.558 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.558 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:51.559 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.559 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:51.559 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.559 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.559 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.560 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.560 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.560 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.561 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.561 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:51.561 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:51.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.563 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:51.564 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.564 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:51.564 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.565 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.565 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.565 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.565 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.566 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.566 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.566 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:51.567 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:51.567 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.567 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.568 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:51.568 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.568 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:51.569 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.569 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.569 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.569 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.569 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.570 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.570 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.571 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:51.571 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:51.571 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.571 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.572 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:51.573 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.573 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:51.574 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.574 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.574 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.574 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.575 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.575 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.576 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.576 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:51.577 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:51.577 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.577 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.578 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:51.578 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.578 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:51.579 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.579 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.579 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.579 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.580 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.580 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.581 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.581 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:11:51.581 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:51.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.583 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:51.583 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.584 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:51.584 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.584 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.585 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.585 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.586 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.586 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.587 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.587 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:51.588 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:51.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.589 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:51.590 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.590 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:51.590 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.591 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.591 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.592 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.592 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.592 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.593 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.593 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:51.593 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:51.593 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:51.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.595 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:51.596 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.596 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.596 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.596 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.597 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.597 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.598 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.598 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000044.121.8", 2020-07-17T12:11:51.598 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:11:51.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.599 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:51.600 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.600 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:51.600 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.600 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.600 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.601 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.601 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.617 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.618 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.618 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:51.618 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:51.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.619 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:51.620 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:51.620 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:51.620 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:51.620 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:51.620 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:51.621 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:51.621 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:51.621 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:51.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.621 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:51.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:51.622 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:51.622 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:51.622 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:51.623 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.623 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.623 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:51.623 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:11:51.624 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000046.126.9 target=00000000044.121.8 2020-07-17T12:11:56.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:56.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:11:56.627 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:11:56.671 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:11:56.672 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:11:56.879 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:11:56.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.880 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:11:56.880 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.880 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:56.880 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.880 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.881 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.881 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.881 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.881 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.882 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.882 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:56.882 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:56.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.883 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:11:56.883 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.883 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:56.884 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.884 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.884 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.884 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.884 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.885 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.885 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.885 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:56.886 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:56.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.886 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:11:56.887 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.887 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:56.887 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.887 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.887 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.888 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.888 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.888 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.889 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.889 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:56.889 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:56.889 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.890 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.890 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:11:56.890 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.890 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:56.891 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.891 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.891 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.891 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.891 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.892 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.892 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.892 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:56.892 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:56.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.894 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:11:56.894 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.894 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:56.895 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.895 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.895 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.895 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.895 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.896 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.898 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.899 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:56.899 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:56.899 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.899 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.900 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:11:56.900 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.900 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:56.900 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.900 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.901 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.901 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.901 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.902 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.903 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.903 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:11:56.903 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:11:56.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.904 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:11:56.904 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.904 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:56.904 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.905 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.905 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.905 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.905 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:56.906 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:11:56.907 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.908 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.909 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.910 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:11:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:11:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.911 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.912 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000044.121.8", 2020-07-17T12:11:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:11:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:11:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.914 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.915 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.915 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.915 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.943 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.943 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.943 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:56.944 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:56.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:11:56.944 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:11:56.945 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:11:56.945 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:11:56.945 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:11:56.945 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:11:56.945 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:11:56.946 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:11:56.946 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:11:56.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.946 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:11:56.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:11:56.947 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:11:56.947 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:11:56.947 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:11:56.947 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.947 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.947 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:11:56.948 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:11:56.949 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000046.126.9 target=00000000044.121.8 2020-07-17T12:12:01.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:01.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:01.952 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:01.998 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:01.998 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:02.202 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:12:02.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.202 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:12:02.203 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.203 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:02.203 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.203 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.204 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.204 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.204 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.204 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.205 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.205 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:02.205 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:02.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.206 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.206 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:12:02.206 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.207 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:02.207 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.207 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.207 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.207 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.207 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.208 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.208 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.208 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:02.209 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:02.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.210 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:12:02.210 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.210 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:02.210 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.210 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.211 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.211 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.211 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.211 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.212 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.212 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:02.212 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:02.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.213 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:12:02.213 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.214 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:02.214 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.214 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.214 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.215 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.215 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.215 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.216 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.216 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:02.216 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:02.216 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.217 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.217 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.217 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:12:02.217 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.218 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:02.218 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.218 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.218 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.218 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.219 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.219 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.219 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.220 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:02.220 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:02.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.221 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:12:02.221 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.221 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:12:02.221 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.221 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.222 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.222 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.222 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.222 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.223 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.223 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:12:02.223 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:02.223 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.223 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.224 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:12:02.224 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.224 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:02.225 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.225 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.225 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.225 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.225 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.226 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.226 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.226 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:02.227 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:02.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.228 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:12:02.228 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.228 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:02.228 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.228 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.228 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.229 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.229 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.229 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.230 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.230 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:02.230 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:02.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:12:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:12:02.231 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000044.121.8", 2020-07-17T12:12:02.233 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:12:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:12:02.234 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.235 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:02.235 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.235 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.235 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.235 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.236 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.270 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.270 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.270 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:02.271 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:02.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:02.272 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:12:02.272 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:02.272 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:02.273 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:02.273 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:02.273 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:02.273 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:02.274 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:02.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.274 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:02.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:02.275 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:02.275 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:02.275 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:02.275 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.275 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.276 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:02.276 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:12:02.278 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000046.126.9 target=00000000044.121.8 2020-07-17T12:12:07.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:07.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:07.281 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:07.326 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:07.327 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:07.526 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:12:07.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.528 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:12:07.528 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.529 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:07.529 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.529 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.530 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.530 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.530 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.533 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.534 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.534 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:07.535 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:07.535 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.535 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.537 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:12:07.537 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.538 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:07.538 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.539 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.539 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.539 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.540 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.540 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.541 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.541 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:07.542 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:07.543 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.543 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.544 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:12:07.544 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.544 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:07.544 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.544 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.545 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.545 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.545 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.545 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.546 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.546 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:07.546 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:07.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.547 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.547 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:12:07.547 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.548 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:07.548 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.548 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.548 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.548 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.548 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.549 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:12:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.551 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:12:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:12:07.554 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.555 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.14.9", 2020-07-17T12:12:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:07.556 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:12:07.557 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.558 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.559 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.567 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:12:07.567 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.567 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:07.567 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.567 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.568 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.568 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.568 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.568 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.569 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.569 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:07.569 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:07.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.571 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:12:07.571 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.571 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:12:07.574 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.574 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.574 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000046.126.9", 2020-07-17T12:12:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:28.105198Z" 2020-07-17T12:12:07.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.577 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.578 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:12:07.578 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.579 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:07.579 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.579 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.579 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.579 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.579 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:12:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:07.596 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:07.597 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.599 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:12:07.601 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-19 2020-07-17T12:12:07.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:07.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:07.602 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:07.693 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:07.919 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:12:07.919 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:12:07.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.920 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:12:07.920 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:07.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.920 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.921 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:12:07.921 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:07.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.921 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:12:07.922 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:07.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.922 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:12:07.922 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:07.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.923 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:12:07.923 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:07.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.924 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:12:07.924 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.12.9" 2020-07-17T12:12:07.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.924 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:12:07.925 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:07.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.925 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:12:07.925 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:07.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.926 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:12:07.927 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000046.146.9" 2020-07-17T12:12:07.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.927 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:12:07.927 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:07.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:12:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:07.928 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:12:07.929 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:12:07.930 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-19 2020-07-17T12:12:07.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:07.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:07.931 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:07.976 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:07.976 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:08.172 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:08.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.173 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:08.173 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.174 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:08.174 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.174 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.175 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:08.175 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:08.175 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.176 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:08.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.176 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.176 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:08.177 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:08.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.178 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:08.178 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.179 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:08.180 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.180 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.181 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:08.181 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:08.182 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.182 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:08.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.183 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.183 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:08.184 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:08.184 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.185 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:08.186 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.186 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:08.186 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.187 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.187 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:08.187 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:08.188 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.188 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:08.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.188 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.189 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:08.189 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:08.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.192 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:08.192 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.193 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:08.193 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.193 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.193 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:08.194 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:08.194 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.194 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:08.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.194 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.195 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:08.195 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:08.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.196 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:08.196 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.196 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:08.196 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.196 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.197 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:08.197 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:08.199 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.201 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:08.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.202 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.202 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:08.202 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:08.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.203 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:08.203 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.204 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:08.204 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.204 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.204 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:08.204 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:08.204 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.205 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:08.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.205 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.205 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:12:08.206 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:08.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.207 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:08.207 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.207 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:08.207 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.207 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.207 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:08.208 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:08.208 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.208 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:08.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.209 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.209 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:08.209 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:08.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.210 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:08.210 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.210 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:08.211 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.212 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.212 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:08.212 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:08.212 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.213 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:08.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.213 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.213 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:08.214 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:08.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.216 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:08.216 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.216 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:08.216 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.217 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.217 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:08.217 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "J9-2-ColLdd3XMhJx2MfpsW2osTdo9z", 2020-07-17T12:12:08.217 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.218 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:08.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.218 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.218 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000045.144.9", 2020-07-17T12:12:08.218 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:12:08.219 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.219 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.219 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:08.219 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.220 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:08.220 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.220 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.222 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:08.222 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:08.222 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.223 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:08.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.224 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.224 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:08.224 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:08.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:08.225 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:08.225 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:08.225 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:08.225 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:08.226 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:08.226 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:08.226 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:08.226 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:08.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.227 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:08.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:08.228 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:08.228 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:08.228 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:08.228 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:08.229 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:08.229 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000046.146.9 target=00000000045.144.9 2020-07-17T12:12:13.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:13.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:13.232 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:13.271 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:13.464 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:13.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.465 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:13.465 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.466 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:13.466 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.467 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.467 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:13.467 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:13.468 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.468 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:13.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.469 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.469 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:13.470 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:13.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.472 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:13.472 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.472 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:13.473 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.473 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.473 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:13.473 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:13.474 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.474 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:13.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.474 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.475 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:13.475 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:13.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.476 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:13.476 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.476 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:13.477 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.477 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.477 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:13.477 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:13.478 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.478 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:13.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.479 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.479 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:13.479 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:13.479 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.481 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:13.481 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.481 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:13.481 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.481 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.482 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:13.482 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:13.482 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.482 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:13.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.483 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.483 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:13.483 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:13.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:13.485 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.485 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.485 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:13.485 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:13.485 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.486 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:13.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.486 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.486 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:13.487 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:13.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.487 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:13.488 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.488 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:13.488 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.488 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.488 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:13.489 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:13.489 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.490 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:13.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.490 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.490 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:12:13.490 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:13.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.493 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:13.493 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.493 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:13.493 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.494 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.494 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:13.494 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:13.494 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.494 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:13.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.495 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.495 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:13.495 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:13.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.496 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:13.497 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.497 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:13.498 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.498 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.498 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:13.498 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:13.498 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.499 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:13.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.499 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.500 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:13.500 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:13.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.501 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:13.501 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.501 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:13.501 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.502 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.502 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:13.502 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "J9-2-ColLdd3XMhJx2MfpsW2osTdo9z", 2020-07-17T12:12:13.504 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.504 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:13.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.506 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.506 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000045.144.9", 2020-07-17T12:12:13.506 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:12:13.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.507 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:13.508 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.508 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:13.508 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.508 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.509 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:13.510 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:13.510 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.511 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:13.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.511 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.511 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:13.512 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:13.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:13.513 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:13.513 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:13.514 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:13.514 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:13.514 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:13.514 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:13.514 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:13.515 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:13.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.515 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:13.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:13.516 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:13.516 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:13.516 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:13.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:13.517 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:13.518 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000046.146.9 target=00000000045.144.9 2020-07-17T12:12:18.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:18.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:18.521 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:18.564 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:18.565 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:18.770 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:18.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.771 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:18.771 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.772 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:18.772 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.772 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:18.773 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:18.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.775 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.775 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:18.775 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:18.777 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.778 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:18.779 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:18.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:18.781 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:18.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.787 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:18.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.799 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:18.800 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.800 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:18.800 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.801 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.801 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:18.801 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:18.802 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.802 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:18.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.803 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.804 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:18.804 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:18.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.806 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:18.806 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.806 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:18.807 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.807 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.807 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:18.808 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:18.808 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.809 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:18.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.809 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.810 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:18.810 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:18.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.812 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:18.812 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.815 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:18.815 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.815 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.816 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:18.816 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:18.816 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.817 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:18.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.817 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.818 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:12:18.818 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:18.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.820 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:18.820 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.820 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:18.821 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.821 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.821 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:18.822 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:18.822 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.823 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:18.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.823 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.823 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:18.824 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:18.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.826 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:18.826 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.827 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:18.827 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.827 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.828 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:18.828 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:18.828 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.829 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:18.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.830 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.830 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:18.830 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:18.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.832 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:18.833 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.833 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:18.833 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.834 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.834 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:18.835 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "J9-2-ColLdd3XMhJx2MfpsW2osTdo9z", 2020-07-17T12:12:18.835 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.836 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:18.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.836 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.837 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000045.144.9", 2020-07-17T12:12:18.837 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:12:18.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.839 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:18.839 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.840 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:18.840 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.840 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.842 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:18.842 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:18.842 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.843 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:18.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.844 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.844 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:18.844 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:18.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:18.846 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:18.847 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:18.847 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:18.847 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:18.848 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:18.848 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:18.848 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:18.849 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:18.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.849 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:18.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:18.850 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:18.850 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:18.851 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:18.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:18.852 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:18.853 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000046.146.9 target=00000000045.144.9 2020-07-17T12:12:23.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:23.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:23.857 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:23.901 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:23.901 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:24.104 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:24.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.104 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:24.104 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.105 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:24.105 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.105 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.106 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:24.106 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:24.107 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.107 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:24.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.108 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.108 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:24.109 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:24.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.110 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:24.110 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.110 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:24.111 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.111 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.111 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:24.111 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:24.111 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.112 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:24.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.112 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.113 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:24.113 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:24.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.114 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:24.114 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:24.116 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:24.116 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.116 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:24.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.117 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.117 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:24.117 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:24.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.118 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:24.118 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.118 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:24.119 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.119 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.119 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:24.119 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:24.120 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.120 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:24.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.120 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.120 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:24.121 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:24.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.122 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:24.122 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.122 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:24.122 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.122 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.123 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:24.123 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:24.124 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.124 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:24.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.124 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.124 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:24.125 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:24.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.126 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:24.126 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.126 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:24.126 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.126 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.127 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:24.127 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:24.127 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.127 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:24.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.128 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.128 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:12:24.128 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:24.128 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.130 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:24.130 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.131 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:24.131 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.131 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.131 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:24.131 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:24.131 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.132 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:24.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.132 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.132 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:24.133 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:24.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.133 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:24.134 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.134 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:24.134 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.135 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.135 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:24.135 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:24.135 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.136 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:24.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.136 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.136 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:24.137 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:24.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.138 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:24.138 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.138 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:24.138 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.138 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.139 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:24.139 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "J9-2-ColLdd3XMhJx2MfpsW2osTdo9z", 2020-07-17T12:12:24.140 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.140 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:24.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.141 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.141 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000045.144.9", 2020-07-17T12:12:24.141 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:12:24.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.142 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.143 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:24.143 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.143 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:24.144 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.144 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.145 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:24.145 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:24.146 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.146 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:24.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.147 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.147 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:24.147 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:24.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.148 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:24.148 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:24.148 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:24.149 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:24.149 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:24.149 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:24.149 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:24.149 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:24.150 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:24.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.150 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:24.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:24.150 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:24.150 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:24.151 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:24.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:24.151 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:24.152 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000046.146.9 target=00000000045.144.9 2020-07-17T12:12:29.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:29.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:29.155 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:29.203 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:29.204 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:29.384 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:29.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.385 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:29.385 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.385 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:29.386 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.386 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.386 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:29.386 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:29.387 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.387 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:29.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.387 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.388 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:29.388 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:29.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.389 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:29.389 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.389 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:29.390 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.390 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.390 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:29.391 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:29.391 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.391 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:29.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.392 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.392 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:29.392 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:29.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.393 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.393 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:29.394 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.394 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:29.394 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.394 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.394 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:29.395 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:29.395 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.395 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:29.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.396 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.396 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:29.396 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:29.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.397 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:29.397 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.397 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:29.398 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.398 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.398 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:29.398 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:29.398 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.399 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:29.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.399 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.399 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:29.400 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:29.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.401 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:29.402 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.402 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:29.402 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.402 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.402 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:29.403 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:29.403 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.403 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:29.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.404 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.404 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:29.404 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:29.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.405 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:29.405 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.405 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:29.406 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.406 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.406 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:29.406 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:29.407 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.407 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:29.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.408 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.408 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:12:29.408 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:29.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.409 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:29.409 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.409 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:29.410 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.410 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.410 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:29.411 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:29.411 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.411 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:29.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.412 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.412 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:29.412 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:29.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.416 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.416 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:29.416 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:29.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.417 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:29.417 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.418 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:29.418 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.418 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.418 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:29.418 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "J9-2-ColLdd3XMhJx2MfpsW2osTdo9z", 2020-07-17T12:12:29.419 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.419 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:29.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.419 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.419 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000045.144.9", 2020-07-17T12:12:29.420 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:12:29.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.421 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:29.421 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.422 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:29.422 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.422 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.424 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:29.424 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:29.424 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.425 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:29.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.426 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.426 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:29.426 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:29.427 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.427 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:29.428 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:29.428 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:29.429 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:29.429 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:29.429 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:29.429 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:29.430 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:29.430 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:29.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.430 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:29.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:29.431 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:29.432 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:29.432 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:29.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:29.433 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:29.433 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000046.146.9 target=00000000045.144.9 2020-07-17T12:12:34.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:34.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:34.437 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:34.483 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:34.483 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:34.675 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:34.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.676 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:34.676 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.676 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:34.676 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.676 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.677 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:34.677 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:34.677 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.678 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:34.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.678 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.678 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:34.678 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:34.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.679 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:34.680 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.680 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:34.680 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.680 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.680 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:34.681 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:34.681 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.681 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:34.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.682 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.682 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:34.682 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:34.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.683 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:34.683 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.684 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:34.684 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.684 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.684 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:34.684 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:34.685 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.685 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:34.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.686 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.686 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:34.686 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:34.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.687 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:34.687 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.687 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:34.688 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.688 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.688 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:34.688 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:34.688 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.689 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:34.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.689 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.689 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:34.690 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:34.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.691 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:34.691 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.691 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:34.691 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.691 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.692 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:34.692 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:34.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.695 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:34.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.695 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.695 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:34.695 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:34.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.696 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:34.697 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.697 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:34.697 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.698 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.698 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:34.698 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:34.699 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.699 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:34.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.699 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.700 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:12:34.700 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:34.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.701 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:34.701 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.701 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:34.701 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.702 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.702 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:34.702 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:34.702 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.703 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:34.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.703 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.703 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:34.704 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:34.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.705 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:34.705 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.705 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:34.705 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.706 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.706 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:34.706 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:34.706 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.707 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:34.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.707 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.707 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.709 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:34.709 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.709 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:34.709 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.710 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.710 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "J9-2-ColLdd3XMhJx2MfpsW2osTdo9z", 2020-07-17T12:12:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:34.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000045.144.9", 2020-07-17T12:12:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:12:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.713 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:34.714 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.714 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.715 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:34.715 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.716 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:34.718 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:34.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:34.720 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:34.721 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:34.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:34.722 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:34.722 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000046.146.9 target=00000000045.144.9 2020-07-17T12:12:39.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:39.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:39.726 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:39.782 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:39.783 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:39.973 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:39.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:39.974 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:39.975 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:39.975 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:39.975 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:39.976 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:39.976 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:39.976 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:39.977 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:39.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.977 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:39.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.978 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:39.979 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:39.979 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:39.979 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:39.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:39.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:39.981 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:39.981 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:39.982 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:39.982 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:39.982 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:39.983 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:39.983 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:39.983 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:39.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.984 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:39.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.984 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:39.985 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:39.985 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:39.985 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:39.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:39.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:39.987 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:39.987 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:39.987 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:39.988 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:39.988 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:39.988 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:39.989 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:39.989 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:39.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.990 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:39.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.990 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:39.991 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:39.991 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:39.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:39.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:39.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:39.992 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:39.993 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:39.993 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:39.993 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:39.994 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:39.994 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:39.994 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:39.995 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:39.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.995 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:39.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.996 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:39.996 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:39.997 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:39.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:39.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:39.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:39.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:39.999 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:39.999 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:39.999 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:40.000 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:40.000 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:40.000 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:40.001 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:40.001 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:40.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.002 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:40.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.002 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:40.003 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:40.003 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:40.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:40.005 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:40.005 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:40.005 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:40.006 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:40.006 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:40.006 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:40.007 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:40.007 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:40.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.007 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:40.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.008 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:40.008 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:12:40.009 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:40.009 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.009 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:40.010 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:40.011 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:40.011 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:40.011 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:40.012 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:40.012 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:40.012 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:40.013 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:40.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.013 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:40.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.014 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:40.014 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:40.014 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:40.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:40.016 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:40.016 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:40.017 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:40.017 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:40.017 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:40.018 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:40.018 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:40.018 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:40.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.019 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:40.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "J9-2-ColLdd3XMhJx2MfpsW2osTdo9z", 2020-07-17T12:12:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000045.144.9", 2020-07-17T12:12:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.151847Z" 2020-07-17T12:12:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:40.041 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:40.042 WARNING:rgw_multi.tests:bucket lbjidp-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000046.146.9 target=00000000045.144.9 2020-07-17T12:12:45.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:45.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:45.045 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:45.088 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:45.089 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:45.261 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:12:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "J9-2-ColLdd3XMhJx2MfpsW2osTdo9z", 2020-07-17T12:12:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000046.146.9", 2020-07-17T12:12:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.056201Z" 2020-07-17T12:12:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.317 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:45.318 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-19 2020-07-17T12:12:45.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:45.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:45.321 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:45.362 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:45.363 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:45.584 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:12:45.585 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:12:45.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.585 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:12:45.586 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:45.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.586 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:12:45.587 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:45.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.587 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:12:45.588 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:45.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.589 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:12:45.589 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:45.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.590 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:12:45.591 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:45.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.591 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:12:45.592 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.12.9" 2020-07-17T12:12:45.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.593 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:12:45.593 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:45.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.594 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:12:45.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:45.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:12:45.596 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000046.146.9" 2020-07-17T12:12:45.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.597 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:12:45.597 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:45.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:45.598 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:45.598 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:12:45.598 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:12:45.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:45.599 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:12:45.599 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:12:45.600 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-19 2020-07-17T12:12:45.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:45.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:45.602 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:45.648 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:45.649 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:45.839 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:45.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.840 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:45.841 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.841 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.841 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.842 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.842 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.842 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.843 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.844 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.844 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.845 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.845 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.847 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:45.847 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.848 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.848 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.848 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.849 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.849 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.849 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.851 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.851 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.852 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.852 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.854 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:45.854 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.855 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.855 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.855 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.856 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.857 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.858 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.858 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.862 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:45.863 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.863 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.863 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.864 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.864 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.864 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.865 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.865 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.866 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.867 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.867 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.869 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:45.869 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.869 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.870 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.870 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.870 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.871 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.872 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.872 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.873 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.874 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:45.875 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.875 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:45.875 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.875 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.876 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:45.876 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:45.876 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.876 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.12.9", 2020-07-17T12:12:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.878 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.879 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.880 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.883 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.883 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.883 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.884 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.884 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.885 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "J9-2-ColLdd3XMhJx2MfpsW2osTdo9z", 2020-07-17T12:12:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:45.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000046.146.9", 2020-07-17T12:12:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.056201Z" 2020-07-17T12:12:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.888 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:45.888 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.889 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.889 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.889 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.890 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.891 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.891 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.891 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.892 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.892 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.893 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:45.894 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:45.895 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:45.895 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:45.895 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:45.895 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:45.896 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:45.896 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:45.896 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:45.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.897 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:45.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:45.898 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:45.898 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:45.898 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:45.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:45.899 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:45.900 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-19 2020-07-17T12:12:45.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:45.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:45.902 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:45.943 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:45.944 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:46.160 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:12:46.161 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:12:46.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.161 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:46.161 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.162 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:46.163 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.164 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:46.164 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.165 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:46.165 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.166 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:46.166 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.15.9" 2020-07-17T12:12:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.168 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:46.169 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.169 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:46.170 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000049.136.9" 2020-07-17T12:12:46.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.170 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:46.171 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.171 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:46.172 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.172 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:12:46.173 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:12:46.174 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-19 2020-07-17T12:12:46.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:46.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:46.176 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:46.261 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:46.262 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:46.461 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:12:46.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.462 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:46.462 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.462 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:46.463 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.463 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.463 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:46.463 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:46.463 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.464 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:46.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.465 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.465 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:46.465 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:46.465 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.467 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:46.467 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.467 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:46.468 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.468 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.468 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:46.468 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:46.468 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.469 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:46.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.469 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.470 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:46.470 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:46.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.472 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:46.472 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.472 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:46.473 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.473 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.473 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:46.474 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:46.476 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.476 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:46.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.476 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.477 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:46.477 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:46.477 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.477 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.478 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:46.478 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.478 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:46.479 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.479 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.479 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:46.479 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:46.479 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.480 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:46.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.480 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.480 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:46.480 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:46.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.481 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:46.482 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.482 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:46.482 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.482 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.482 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:46.483 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:46.483 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.483 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:46.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.483 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.485 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:46.485 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.485 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dummy", 2020-07-17T12:12:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:46.486 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.487 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:12:46.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.488 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.488 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.15.9", 2020-07-17T12:12:46.488 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:46.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.489 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.491 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.493 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.493 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:46.494 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:46.494 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.494 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:46.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.495 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.495 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:46.495 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:46.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.496 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:46.497 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.497 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:46.497 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.497 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.498 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:46.498 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:46.498 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.499 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:46.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.500 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.500 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:46.500 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:46.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.502 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:46.502 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.502 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:12:46.503 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.503 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.503 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:12:46.503 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "MQ.yErR1ERWRUwZuP4MJi-IhHhGeLKQ", 2020-07-17T12:12:46.503 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.504 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 2 2020-07-17T12:12:46.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.504 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.504 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000049.136.9", 2020-07-17T12:12:46.505 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:28.056201Z" 2020-07-17T12:12:46.505 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.505 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.506 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:46.507 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.508 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:46.508 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.508 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.509 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:46.510 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:46.510 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.510 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:46.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.511 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.511 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:46.512 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:46.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.513 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:46.533 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:12:46.533 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:12:46.534 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:12:46.534 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:12:46.534 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:12:46.534 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:12:46.534 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:12:46.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.535 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:12:46.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.535 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:12:46.536 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:12:46.537 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:12:46.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.538 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:12:46.538 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-19 2020-07-17T12:12:46.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:46.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:46.540 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:46.605 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:46.605 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:46.802 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:12:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:12:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:12:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:12:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:12:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:12:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:12:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:12:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.15.9" 2020-07-17T12:12:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:12:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:12:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:12:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000049.136.9" 2020-07-17T12:12:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:12:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:12:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:12:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:12:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:12:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:12:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:12:46.811 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:12:46.812 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-19 2020-07-17T12:12:46.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:46.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:12:46.814 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:12:46.858 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:12:46.858 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:12:47.059 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:12:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.059 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:12:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.060 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.061 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:12:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:47.064 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.064 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.064 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:47.064 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:47.065 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.065 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:47.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.065 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.066 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:47.066 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:47.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.067 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:12:47.067 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.067 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:47.067 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.067 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.068 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:47.068 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:47.068 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.068 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:47.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.069 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.069 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:47.069 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:47.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.070 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:12:47.070 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.070 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:47.070 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.071 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.071 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:47.071 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:47.071 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.072 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:47.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.072 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.072 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:47.072 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:47.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.073 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:12:47.073 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.075 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:47.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.075 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.075 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:47.076 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:47.076 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.076 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.077 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:12:47.077 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.077 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:12:47.077 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.077 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.078 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dummy", 2020-07-17T12:12:47.078 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "icEpJ4gZP6rNhs7wlNVh6oO5bbytlJt", 2020-07-17T12:12:47.078 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.078 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:12:47.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.079 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.079 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.15.9", 2020-07-17T12:12:47.079 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:44.735610Z" 2020-07-17T12:12:47.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.080 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:12:47.081 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.081 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:47.081 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.081 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.081 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:47.081 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:47.082 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.082 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:47.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.082 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.083 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:47.083 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:47.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.084 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:12:47.084 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.084 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:47.084 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.084 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.085 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:47.085 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:47.085 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.085 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:47.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.086 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.086 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:47.086 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:47.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.087 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:12:47.087 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.087 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:12:47.087 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.088 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.088 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:12:47.088 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "MQ.yErR1ERWRUwZuP4MJi-IhHhGeLKQ", 2020-07-17T12:12:47.088 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.088 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 2 2020-07-17T12:12:47.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.089 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.089 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000049.136.9", 2020-07-17T12:12:47.089 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:28.056201Z" 2020-07-17T12:12:47.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.090 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:12:47.090 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.091 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:47.091 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.091 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.092 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:47.092 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:47.092 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:47.093 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:47.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:12:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:12:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:12:47.095 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:12:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:12:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:12:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:12:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:12:47.096 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:12:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:12:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:12:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:12:47.097 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:12:47.098 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:12:47.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:12:47.098 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:12:47.099 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-19 2020-07-17T12:12:47.099 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-19 zones={test-zone2, test-zone1} 2020-07-17T12:12:47.116 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:12:47.127 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:12:47.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.148 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:12:47.158 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:12:47.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:47.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:12:47.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:12:47.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:12:47.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:47.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:47.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:47.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:47.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:47.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:47.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:47.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:48.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:49.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:50.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:51.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:52.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:53.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:54.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:55.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:56.111 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-19 zones={test-zone2, test-zone1} 2020-07-17T12:12:56.112 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-19 zones={test-zone3, test-zone1} 2020-07-17T12:12:56.121 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:12:56.127 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:12:56.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.142 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:12:56.151 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:12:56.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:12:56.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:12:56.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:12:56.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:12:56.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:56.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:56.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:56.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:56.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:56.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:56.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:56.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:57.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:58.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:12:59.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:00.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:01.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:02.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:03.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:04.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:05.054 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-19 zones={test-zone3, test-zone1} 2020-07-17T12:13:05.055 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-19 zones={test-zone3, test-zone2} 2020-07-17T12:13:05.067 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:13:05.075 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:13:05.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.089 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:13:05.096 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-17T12:13:05.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:05.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:13:05.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:13:05.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-17T12:13:05.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:05.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:05.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:05.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:05.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:05.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:05.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:05.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:06.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:07.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:08.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:09.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:09.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:09.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:09.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:09.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:09.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:09.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:09.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:09.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:10.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:11.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:12.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:13.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:14.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:14.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-19 zones={test-zone3, test-zone2} 2020-07-17T12:13:14.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:14.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:14.210 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:14.252 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:14.252 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:14.490 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:13:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:13:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:13:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:13:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:13:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987926.483526_218.1", 2020-07-17T12:13:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:12:06.483526Z" 2020-07-17T12:13:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:14.495 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:14.496 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-17T12:13:14.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:14.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:14.498 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:14.539 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:14.539 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:14.766 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:14.767 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:13:14.767 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:13:14.767 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:13:14.767 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:13:14.768 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 14149894388601651757 2020-07-17T12:13:14.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:14.768 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:14.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:14.768 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:14.768 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:14.769 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:14.769 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:13:14.769 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:14.769 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:14.769 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:14.770 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:13:14.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:14.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:14.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:14.770 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:14.771 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:14.771 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:14.771 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:14.771 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:14.771 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:14.771 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:14.772 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:14.772 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:14.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:14.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:14.772 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:14.773 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:14.773 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:14.773 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:13:14.773 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:14.773 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:14.773 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:14.774 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:13:14.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:14.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:14.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:14.774 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:14.775 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:14.775 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:14.775 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987926.483526_218.1", 2020-07-17T12:13:14.775 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:14.775 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:14.776 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:14.776 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:12:06.483526Z" 2020-07-17T12:13:14.776 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:14.776 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:14.777 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:14.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:14.777 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:13:14.777 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2020-07-17T12:13:14.777 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2020-07-17T12:13:14.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:14.778 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:14.779 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-17T12:13:14.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:14.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:14.782 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:14.833 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:14.833 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:15.048 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:15.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.049 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:13:15.049 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:13:15.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.050 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:15.050 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:13:15.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.051 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:13:15.052 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:13:15.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.053 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987926.483526_218.1", 2020-07-17T12:13:15.053 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:12:06.483526Z" 2020-07-17T12:13:15.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:15.054 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:15.055 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-17T12:13:15.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:15.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:15.058 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:15.151 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:15.152 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:15.373 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:15.373 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:13:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:13:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:13:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:13:15.374 INFO:teuthology.orchestra.run.smithi146.stdout: "instance_id": 10221300329629434139 2020-07-17T12:13:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:15.375 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:13:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:15.376 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:13:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:15.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:15.378 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:15.379 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:15.380 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:13:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:13:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987926.483526_218.1", 2020-07-17T12:13:15.383 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:12:06.483526Z" 2020-07-17T12:13:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:15.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:13:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 0, 2020-07-17T12:13:15.385 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 0 2020-07-17T12:13:15.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:15.386 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:15.387 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-17T12:13:15.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:15.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:15.389 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:15.469 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:15.470 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:15.687 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:15.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.687 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:13:15.687 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:13:15.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.688 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:15.688 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:13:15.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.689 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:13:15.689 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:13:15.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:15.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:15.690 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987926.483526_218.1", 2020-07-17T12:13:15.690 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:12:06.483526Z" 2020-07-17T12:13:15.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:15.690 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:15.691 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-17T12:13:15.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:15.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:15.693 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:15.736 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:15.736 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:16.106 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:13:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:13:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:13:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:13:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 3210402864628377302 2020-07-17T12:13:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:16.109 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:13:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:16.110 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:13:16.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:16.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:16.113 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:16.114 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:16.115 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:13:16.116 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:16.117 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:13:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:16.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987926.483526_218.1", 2020-07-17T12:13:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:12:06.483526Z" 2020-07-17T12:13:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:13:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2020-07-17T12:13:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2020-07-17T12:13:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:16.131 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:16.132 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-17T12:13:16.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:16.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:16.134 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:16.213 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:16.214 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:16.443 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:16.443 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:16.444 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.524415_146.1", 2020-07-17T12:13:16.444 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:11:22.524415Z" 2020-07-17T12:13:16.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:16.445 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:16.445 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:16.445 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "0.000000" 2020-07-17T12:13:16.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:16.446 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:16.446 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.617050_107.1", 2020-07-17T12:13:16.447 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:10:10.617050Z" 2020-07-17T12:13:16.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:16.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:16.448 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987948.536498_242.1", 2020-07-17T12:13:16.448 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:12:28.536498Z" 2020-07-17T12:13:16.448 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:16.449 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:16.450 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-17T12:13:16.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:16.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:16.453 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:16.496 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:16.730 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:16.731 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:13:16.731 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:13:16.731 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:13:16.732 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:13:16.732 INFO:teuthology.orchestra.run.smithi146.stdout: "instance_id": 7737347243854991200 2020-07-17T12:13:16.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:16.733 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:16.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:16.733 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:16.733 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:16.734 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:16.734 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.524415_146.1", 2020-07-17T12:13:16.734 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:16.735 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:16.735 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:16.735 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:22.524415Z" 2020-07-17T12:13:16.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:16.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:16.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:16.737 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:16.737 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:16.737 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:16.738 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:16.739 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:16.739 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:16.739 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:16.739 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:16.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:16.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:16.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:16.740 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:16.741 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:16.741 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:16.741 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.617050_107.1", 2020-07-17T12:13:16.742 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:16.742 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:16.742 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:16.742 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.617050Z" 2020-07-17T12:13:16.743 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:16.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:16.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:16.743 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:16.743 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:16.743 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:16.744 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987948.536498_242.1", 2020-07-17T12:13:16.744 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:16.744 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:16.744 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:16.745 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:12:28.536498Z" 2020-07-17T12:13:16.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:16.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:16.745 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:16.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:16.745 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:13:16.746 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 0, 2020-07-17T12:13:16.746 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 0 2020-07-17T12:13:16.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:16.746 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:16.747 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-17T12:13:16.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:16.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:16.749 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:16.789 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:16.790 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:17.015 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:17.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:17.016 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.524415_146.1", 2020-07-17T12:13:17.016 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:11:22.524415Z" 2020-07-17T12:13:17.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:17.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:17.017 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:17.017 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "0.000000" 2020-07-17T12:13:17.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:17.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:17.017 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.617050_107.1", 2020-07-17T12:13:17.017 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:10:10.617050Z" 2020-07-17T12:13:17.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:17.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:17.018 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987948.536498_242.1", 2020-07-17T12:13:17.018 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:12:28.536498Z" 2020-07-17T12:13:17.018 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:17.019 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:17.019 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-17T12:13:17.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:17.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:17.021 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:17.064 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:17.064 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:17.271 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:17.271 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:13:17.271 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:13:17.272 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:13:17.272 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:13:17.272 INFO:teuthology.orchestra.run.smithi146.stdout: "instance_id": 7555103532793312835 2020-07-17T12:13:17.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.272 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:17.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.273 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:17.273 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:17.273 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:17.273 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.524415_146.1", 2020-07-17T12:13:17.274 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:17.274 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:17.274 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:17.274 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:22.524415Z" 2020-07-17T12:13:17.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.275 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:17.275 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:17.275 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:17.276 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:17.276 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:17.276 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:17.276 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:17.276 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:17.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.277 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:17.278 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:17.278 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:17.278 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.617050_107.1", 2020-07-17T12:13:17.278 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:17.279 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:17.279 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:17.279 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.617050Z" 2020-07-17T12:13:17.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.280 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:17.280 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:17.280 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:17.280 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987948.536498_242.1", 2020-07-17T12:13:17.281 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:17.281 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:17.281 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:17.281 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:12:28.536498Z" 2020-07-17T12:13:17.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.282 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:17.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.282 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:13:17.283 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 0, 2020-07-17T12:13:17.283 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 0 2020-07-17T12:13:17.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.283 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:17.284 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-17T12:13:17.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:17.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:17.286 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:17.365 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:17.366 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:17.605 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:17.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.605 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.540719_108.1", 2020-07-17T12:13:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.540719Z" 2020-07-17T12:13:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:13:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.854745_76.1", 2020-07-17T12:13:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.854745Z" 2020-07-17T12:13:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987948.553096_194.1", 2020-07-17T12:13:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:12:28.553096Z" 2020-07-17T12:13:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.610 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:17.611 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-17T12:13:17.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:17.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:17.614 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:17.700 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:17.700 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:17.923 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:17.923 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:13:17.924 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:13:17.924 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:13:17.924 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:13:17.925 INFO:teuthology.orchestra.run.smithi146.stdout: "instance_id": 3231250023779715025 2020-07-17T12:13:17.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.925 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:17.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.926 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:17.926 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:17.926 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:17.927 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.540719_108.1", 2020-07-17T12:13:17.927 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:17.927 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:17.928 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:17.928 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:22.540719Z" 2020-07-17T12:13:17.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.929 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:17.929 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:17.930 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:17.930 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:17.931 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:17.931 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:17.931 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:17.932 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:17.932 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.933 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:17.933 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:17.933 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:17.934 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.854745_76.1", 2020-07-17T12:13:17.934 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:17.934 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:17.934 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:17.935 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.854745Z" 2020-07-17T12:13:17.935 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:17.936 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:17.937 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:17.937 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:17.937 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987948.553096_194.1", 2020-07-17T12:13:17.937 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:17.938 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:17.938 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:17.938 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:12:28.553096Z" 2020-07-17T12:13:17.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.939 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:17.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:17.940 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:13:17.940 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 0, 2020-07-17T12:13:17.943 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 0 2020-07-17T12:13:17.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:17.943 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:17.945 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-17T12:13:17.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:17.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:17.948 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:18.023 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:18.024 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:18.231 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:18.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:18.231 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.540719_108.1", 2020-07-17T12:13:18.231 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.540719Z" 2020-07-17T12:13:18.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:18.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:18.232 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:18.232 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:13:18.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:18.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:18.233 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.854745_76.1", 2020-07-17T12:13:18.233 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.854745Z" 2020-07-17T12:13:18.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:18.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:18.234 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987948.553096_194.1", 2020-07-17T12:13:18.234 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:12:28.553096Z" 2020-07-17T12:13:18.234 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:18.234 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:18.235 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-17T12:13:18.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:18.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:18.237 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:18.280 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:18.281 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:18.509 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:18.510 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:13:18.510 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:13:18.510 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:13:18.510 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:13:18.510 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 9026716065849677085 2020-07-17T12:13:18.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:18.511 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:18.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:18.511 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:18.511 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:18.512 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:18.512 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.540719_108.1", 2020-07-17T12:13:18.512 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:18.512 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:18.512 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:18.512 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:22.540719Z" 2020-07-17T12:13:18.513 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:18.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:18.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:18.513 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:18.513 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:18.514 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:18.514 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:18.514 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:18.514 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:18.515 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:18.515 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:18.515 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:18.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:18.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:18.515 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:18.516 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:18.516 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:18.516 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.854745_76.1", 2020-07-17T12:13:18.516 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:18.516 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:18.517 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:18.517 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.854745Z" 2020-07-17T12:13:18.517 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:18.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:18.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:18.518 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:18.518 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:18.518 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:18.518 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987948.553096_194.1", 2020-07-17T12:13:18.519 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:18.519 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:18.519 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:18.519 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:12:28.553096Z" 2020-07-17T12:13:18.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:18.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:18.520 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:18.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:18.520 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:13:18.520 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2020-07-17T12:13:18.520 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2020-07-17T12:13:18.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:18.521 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:18.522 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-17T12:13:18.523 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-07-17T12:13:18.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:18.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:18.563 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:18.601 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:18.601 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:18.795 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:13:18.827 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:18.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:18.828 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987998.548993_99.1", 2020-07-17T12:13:18.828 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:13:18.548993Z" 2020-07-17T12:13:18.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:18.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:18.829 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:13:18.829 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:13:18.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:18.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:18.830 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:18.831 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:13:18.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:18.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:18.832 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:18.833 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:13:18.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:18.833 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:18.835 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:13:18.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:18.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:18.837 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:18.878 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:18.878 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:19.080 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:19.081 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:13:19.081 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:13:19.082 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:13:19.082 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:13:19.082 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:13:19.083 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.083 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:19.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:19.084 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:19.084 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:19.085 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:19.085 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987998.548993_99.1", 2020-07-17T12:13:19.086 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:19.086 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:13:19.086 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:19.087 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.548993Z", 2020-07-17T12:13:19.087 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:13:19.089 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:19.089 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:19.089 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:19.089 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:13:19.089 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:19.091 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:19.091 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:19.091 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:19.091 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:19.092 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:19.092 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:19.092 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:13:19.092 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:19.097 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:19.097 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:19.097 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:19.098 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:19.098 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:19.098 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:19.098 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:19.099 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:13:19.099 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.099 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:19.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.100 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:13:19.100 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:13:19.100 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:13:19.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.101 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:19.103 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:13:19.103 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:13:19.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:19.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:19.105 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:19.148 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:19.149 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:19.364 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:13:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:13:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:13:19.365 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:13:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:13:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:19.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:13:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987998.548993_99.1", 2020-07-17T12:13:19.367 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:13:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:18.548993Z", 2020-07-17T12:13:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:19.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:13:19.369 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:13:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:13:19.370 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:19.371 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:13:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:19.372 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:13:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:19.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:13:19.374 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:13:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:19.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:13:19.376 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:13:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:13:19.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:19.377 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:19.378 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:13:19.378 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:13:19.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:19.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:19.380 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:19.422 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:19.423 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:19.693 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:19.693 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:13:19.694 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:13:19.694 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:13:19.694 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:13:19.695 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:13:19.695 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.695 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:19.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:19.696 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:19.696 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:19.696 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:19.697 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987998.548993_99.1", 2020-07-17T12:13:19.697 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:19.697 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:13:19.697 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:19.698 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.548993Z", 2020-07-17T12:13:19.698 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.698 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:19.698 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:19.699 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:19.699 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:19.699 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:13:19.699 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:19.699 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:19.700 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:19.700 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:13:19.700 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:19.701 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:19.701 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:19.701 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:19.701 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:19.701 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:19.702 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:19.702 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:19.702 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:13:19.702 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.702 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:19.703 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:19.703 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:19.703 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:19.704 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:19.704 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:19.704 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:19.704 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:19.704 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:13:19.705 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:19.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.705 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:19.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:19.706 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:13:19.706 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:13:19.706 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:13:19.707 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:19.707 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:19.708 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:13:19.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:19.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:19.764 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:19.805 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:19.806 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:20.016 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:20.016 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:20.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.017 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:20.017 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.018 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:20.018 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.019 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:20.020 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.020 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:20.020 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.021 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:20.022 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.022 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:20.022 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.10.5" 2020-07-17T12:13:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.024 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:20.025 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.025 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:20.025 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.026 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:20.026 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:20.027 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:20.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:20.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:20.029 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:20.080 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:20.080 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:20.271 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:13:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:13:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.10.5", 2020-07-17T12:13:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:19.855970Z" 2020-07-17T12:13:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:20.322 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:20.322 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:20.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:20.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:20.324 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:20.368 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:20.369 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:20.602 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:20.602 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:20.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.602 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:20.603 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.603 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:20.603 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.604 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:20.604 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.605 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:20.605 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.605 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:20.606 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.606 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:20.606 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.607 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:20.607 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.608 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:20.608 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.608 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:20.609 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.10.5" 2020-07-17T12:13:20.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.609 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:20.609 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.610 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:20.610 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:20.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.610 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:20.611 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:20.612 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:20.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:20.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:20.613 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:20.916 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:20.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.917 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:20.917 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.917 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.917 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.917 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.918 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.918 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.918 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.918 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.919 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.919 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.919 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.920 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:20.920 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.921 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.921 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.921 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.921 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.921 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.922 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.923 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.923 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.923 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.923 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.924 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.924 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.924 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:20.925 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.925 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.925 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.925 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.925 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.926 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.926 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.926 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.927 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.927 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.927 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.927 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.927 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.928 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:20.928 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.928 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.929 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.929 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.929 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.929 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.931 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.931 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.932 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.932 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.933 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.934 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:20.934 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.934 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.934 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.934 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.935 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.935 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.935 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.935 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.936 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.936 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.936 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.937 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:20.937 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.938 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.938 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.938 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.938 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.938 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.939 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.939 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.939 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.940 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.940 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.941 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:20.941 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.941 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.941 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.942 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.942 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.942 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.942 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.943 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.943 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.944 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.944 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.945 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:20.945 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.945 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.945 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.946 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.946 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.946 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.946 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.947 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.947 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.947 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.947 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.948 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.948 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:20.948 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.949 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:20.949 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.949 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.949 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:13:20.949 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.950 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.950 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:13:20.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.950 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.951 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.10.5", 2020-07-17T12:13:20.951 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:19.849963Z" 2020-07-17T12:13:20.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.952 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:20.952 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.952 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.952 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.953 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.953 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.953 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.953 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.954 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.955 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.955 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:20.957 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:20.957 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:20.957 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:20.957 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.958 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:20.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.960 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.960 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:20.960 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:20.960 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:20.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:20.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:20.962 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:21.038 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:21.039 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:21.259 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:21.260 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:21.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.260 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:21.260 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.261 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:21.261 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.262 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:21.262 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.263 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:21.263 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.264 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:21.264 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.265 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:21.265 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.266 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:21.266 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.267 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:21.268 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.268 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:21.269 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.10.5" 2020-07-17T12:13:21.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.270 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:21.270 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:21.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:21.271 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:21.271 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:21.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:21.272 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:21.272 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:21.273 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:21.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:21.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:21.276 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:21.323 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:21.324 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:21.535 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.10.5", 2020-07-17T12:13:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:19.712236Z" 2020-07-17T12:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.599 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:21.600 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:21.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:21.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:21.602 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:21.687 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:21.687 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:21.921 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.11.5" 2020-07-17T12:13:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:21.930 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:21.931 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:13:21.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:21.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:21.933 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:21.982 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:21.983 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:22.199 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:13:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:13:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.11.5", 2020-07-17T12:13:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:20.142014Z" 2020-07-17T12:13:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:22.258 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.258 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.258 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.258 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.258 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.259 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.259 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.259 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.260 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.260 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.260 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.261 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:22.261 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:13:22.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:22.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:22.263 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:22.305 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:22.306 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:22.535 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:22.535 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:22.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.535 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:22.536 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.536 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:22.536 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.537 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:22.537 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.538 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:22.538 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.538 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:22.539 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.539 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:22.539 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.540 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:22.540 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.541 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:22.541 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.541 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:22.542 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.11.5" 2020-07-17T12:13:22.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.542 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:22.542 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:22.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:22.543 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:22.543 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:22.543 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:22.543 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:22.544 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:22.545 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:13:22.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:22.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:22.546 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:22.591 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:22.591 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.843 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.843 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.843 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.843 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.844 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.844 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.845 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:22.845 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.845 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:22.845 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.845 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.846 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:13:22.846 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.846 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.846 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:13:22.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.847 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.847 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.11.5", 2020-07-17T12:13:22.847 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:20.142039Z" 2020-07-17T12:13:22.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.850 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:22.852 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:22.852 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:22.853 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:22.853 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:22.853 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:22.853 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:22.853 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:22.853 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:22.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.854 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:22.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:22.854 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:22.854 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:22.855 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:22.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:22.856 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:13:22.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:22.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:22.857 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:22.898 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:22.898 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:23.119 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.121 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.122 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:23.123 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:23.128 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.8.5" 2020-07-17T12:13:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:23.129 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.130 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.131 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:23.131 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:23.132 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:13:23.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:23.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:23.137 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:23.226 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:23.227 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:23.431 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:23.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.432 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:23.432 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.433 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.433 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.433 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.434 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.434 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.434 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.435 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.435 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.436 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.436 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.437 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:23.438 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.438 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.438 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.438 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.439 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.439 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.439 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.440 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.440 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.441 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.441 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.443 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:23.443 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.443 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.443 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.444 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.444 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.444 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.445 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.445 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.446 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.446 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.446 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.448 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:23.448 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.448 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.449 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.449 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.449 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.449 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.450 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.450 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.451 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.451 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.451 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.452 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.452 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.453 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:23.454 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.455 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.455 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.455 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.456 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.456 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.456 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.457 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.457 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.458 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.458 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.459 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.459 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:23.459 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.460 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.460 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.460 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.460 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.460 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.460 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.461 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.461 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.461 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.462 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.463 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:23.463 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.463 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.463 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.463 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.463 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.464 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.464 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.465 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.465 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.465 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.465 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.467 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:23.467 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.467 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.467 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.467 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.468 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.468 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.468 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.468 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.469 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.469 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.469 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.469 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.470 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:23.471 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.471 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:23.471 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.471 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.471 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:13:23.471 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.472 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.472 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:13:23.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.472 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.473 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.8.5", 2020-07-17T12:13:23.473 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:20.169937Z" 2020-07-17T12:13:23.473 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.473 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.474 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:23.474 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.474 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.474 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.474 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.475 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.475 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.475 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.475 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.477 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.477 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.477 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.478 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.478 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.478 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:23.479 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:23.479 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:23.479 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:23.479 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:23.479 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:23.480 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:23.480 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:23.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.480 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:23.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.481 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:23.482 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:23.482 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:23.482 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.483 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:23.484 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:13:23.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:23.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:23.487 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:23.557 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:23.557 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:23.782 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:23.783 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:23.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.784 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:23.784 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.785 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:23.785 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.786 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:23.787 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.788 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:23.788 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.789 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:23.790 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.791 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:23.791 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.792 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:23.792 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.793 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:23.793 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.794 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:23.795 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.8.5" 2020-07-17T12:13:23.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.796 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:23.796 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:23.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:23.797 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:23.798 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:23.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:23.798 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:23.798 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:23.799 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:13:23.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:23.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:23.801 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:23.846 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:23.847 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:24.054 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:24.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.055 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:24.055 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.055 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.056 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.056 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.056 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.056 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.056 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.057 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.057 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.057 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.057 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.058 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:24.059 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.059 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.059 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.059 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.060 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.060 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.060 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.060 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.061 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.061 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.061 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.062 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:24.062 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.063 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.063 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.063 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.063 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.063 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.064 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.064 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.064 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.064 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.065 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.066 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:24.066 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.066 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.066 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.066 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.066 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.067 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.067 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.067 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.068 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.068 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.068 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.069 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:24.069 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.069 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.069 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.070 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.070 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.070 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.070 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.071 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.071 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.071 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.071 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.073 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:24.073 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.073 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.073 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.073 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.073 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.074 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.074 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.074 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.075 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.075 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.075 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.076 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:24.076 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.076 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.076 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.077 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.077 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.077 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.077 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.078 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.078 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.078 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.078 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.079 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:24.079 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.080 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.080 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.080 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.080 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.081 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.081 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.082 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.082 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.082 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.083 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.083 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:13:24.085 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.086 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.086 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:13:24.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.086 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.8.5", 2020-07-17T12:13:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:20.167891Z" 2020-07-17T12:13:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.088 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:24.088 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.088 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.089 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.089 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.090 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.090 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.090 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.091 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.093 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.093 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.093 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:24.094 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:24.096 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:24.096 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:24.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.096 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:24.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:24.097 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:24.097 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:24.098 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:24.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:24.099 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:24.100 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:13:24.101 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-20 zones={test-zone2, test-zone1} 2020-07-17T12:13:24.112 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:13:24.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:24.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:13:24.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:24.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:24.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:24.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:24.389 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-20 zones={test-zone2, test-zone1} 2020-07-17T12:13:24.390 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-20 zones={test-zone3, test-zone1} 2020-07-17T12:13:24.399 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:13:24.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:24.450 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:13:24.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:24.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:24.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:24.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:24.661 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-20 zones={test-zone3, test-zone1} 2020-07-17T12:13:24.662 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-20 zones={test-zone3, test-zone2} 2020-07-17T12:13:24.671 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:13:24.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:24.719 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:13:24.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:13:24.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:24.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:24.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:13:24.926 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-20 zones={test-zone3, test-zone2} 2020-07-17T12:13:24.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:24.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:24.947 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:24.986 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:24.986 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:25.162 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:13:25.195 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:25.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:25.195 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988004.936521_101.1", 2020-07-17T12:13:25.195 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:13:24.936521Z" 2020-07-17T12:13:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:13:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:13:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:25.197 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:25.197 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:13:25.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:25.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:25.197 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:25.198 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:13:25.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:25.198 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:25.199 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:13:25.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:25.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:25.201 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:25.242 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:25.242 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:25.472 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:25.473 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:13:25.473 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:13:25.473 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:13:25.474 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:13:25.474 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:13:25.474 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:25.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:25.475 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:25.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:25.475 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:25.475 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:25.476 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:25.476 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988004.936521_101.1", 2020-07-17T12:13:25.476 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:25.476 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:13:25.477 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:25.478 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:24.936521Z", 2020-07-17T12:13:25.478 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:25.478 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:25.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:25.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:25.479 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:25.479 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:25.480 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:25.480 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:13:25.480 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:25.480 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:25.480 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:25.481 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:13:25.481 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:25.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:25.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:25.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:13:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:25.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:25.484 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:13:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:25.488 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:13:25.488 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:13:25.488 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:13:25.489 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:25.489 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:25.491 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:13:25.491 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:13:25.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:25.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:25.493 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:25.535 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:25.765 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:25.766 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:13:25.766 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:13:25.766 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:13:25.766 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:13:25.767 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:13:25.767 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:25.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:25.767 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:25.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:25.768 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:25.768 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:25.768 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:13:25.768 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988004.936521_101.1", 2020-07-17T12:13:25.768 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:25.769 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:13:25.769 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:25.769 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:24.936521Z", 2020-07-17T12:13:25.769 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:25.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:25.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:25.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:25.770 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:25.771 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:25.771 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:13:25.771 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:13:25.771 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:25.771 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:25.772 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:25.772 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:13:25.772 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:25.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:25.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:25.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:25.773 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:25.773 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:25.773 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:13:25.773 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:25.774 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:25.774 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:25.774 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:25.774 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:13:25.774 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:25.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:25.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:25.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:25.776 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:25.776 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:25.777 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:13:25.777 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:13:25.777 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:13:25.777 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:13:25.778 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:13:25.778 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:13:25.778 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:13:25.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:25.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:25.779 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:25.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:25.780 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:13:25.781 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:13:25.781 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:13:25.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:25.781 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:25.783 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:13:25.783 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:13:25.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:25.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:25.785 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:25.825 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:25.826 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:26.097 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:26.097 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:13:26.097 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:13:26.101 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:13:26.101 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:13:26.101 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:13:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988004.936521_101.1", 2020-07-17T12:13:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:13:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:24.936521Z", 2020-07-17T12:13:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:13:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:26.106 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:26.106 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:26.111 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:13:26.111 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.113 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:26.113 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:26.113 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:26.113 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:26.113 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:26.114 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:13:26.114 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:26.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.115 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:26.115 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.115 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:13:26.115 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:13:26.115 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:13:26.115 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:13:26.116 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:13:26.116 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:13:26.116 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:13:26.116 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.116 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.116 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:26.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.117 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:13:26.117 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:13:26.117 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:13:26.117 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.118 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:26.119 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:13:26.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:26.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:26.172 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:26.213 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:26.213 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:26.428 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:26.429 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:26.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.429 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:26.430 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.430 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:26.430 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.431 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:26.431 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.432 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:26.432 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.432 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:26.433 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.433 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:26.433 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.434 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:26.434 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.435 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:26.435 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.435 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:26.435 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.13.12" 2020-07-17T12:13:26.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.436 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:26.436 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:26.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:26.437 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:26.437 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:26.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:26.437 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:26.438 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:26.439 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:26.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:26.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:26.440 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:26.485 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:26.485 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:26.706 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:26.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.707 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:26.707 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.707 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.707 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.707 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.708 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.708 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.708 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.718 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.718 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.718 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.718 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.718 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.719 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.719 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.719 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.719 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.720 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.720 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.720 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:26.721 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.721 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.721 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.721 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.721 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.721 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.722 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.722 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.722 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.723 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.723 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.723 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.723 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.724 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:26.724 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.724 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.724 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.724 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.724 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.726 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.726 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.726 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.726 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.726 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.727 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:26.727 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.727 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.728 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.728 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.728 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.728 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.728 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.729 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.729 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.729 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.730 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.730 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.730 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.731 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:26.731 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.731 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.731 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.731 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.732 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.732 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.732 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.732 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.733 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.733 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.733 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.734 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:26.734 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.734 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:26.734 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.735 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.735 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:13:26.735 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.735 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.736 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:13:26.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.736 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.736 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.10.5", 2020-07-17T12:13:26.736 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:19.855970Z" 2020-07-17T12:13:26.736 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.738 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.738 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.738 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.738 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.738 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.739 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.740 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.740 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.740 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:26.741 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:26.742 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:26.742 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:26.742 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:26.742 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:26.742 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:26.742 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:26.743 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:26.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.743 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:26.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:26.743 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:26.745 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:26.745 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.13.12 target=00000000001.10.5 2020-07-17T12:13:31.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:31.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:31.749 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:31.794 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:31.794 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:32.012 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:32.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.013 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:32.013 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.013 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.014 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.014 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.014 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.014 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.014 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.015 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.015 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.015 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.015 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.016 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.016 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.017 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:32.017 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.017 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.017 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.017 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.018 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.018 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.018 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.018 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.019 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.019 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.019 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.020 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:32.020 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.020 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.021 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.021 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.021 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.021 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.021 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.022 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.023 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.024 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.025 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.026 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.027 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.027 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.027 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.027 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.028 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:32.028 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.028 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.029 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.029 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.029 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.029 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.029 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.030 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.030 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.030 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.031 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.031 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.031 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.032 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:32.032 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.032 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.032 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.032 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.032 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.033 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.033 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.033 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.034 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.034 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.034 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.034 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.034 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.035 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:32.035 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.036 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.036 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.036 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.036 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.036 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.037 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.038 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.038 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.038 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.038 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.038 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.039 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:32.039 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.039 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.039 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.040 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.040 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.040 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.040 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.041 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.041 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.041 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.041 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.041 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.042 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.042 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:32.042 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.043 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:32.043 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.043 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.043 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:13:32.043 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.043 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.044 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:13:32.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.044 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.044 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.10.5", 2020-07-17T12:13:32.045 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:19.855970Z" 2020-07-17T12:13:32.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.045 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:32.046 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.046 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.046 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.046 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.046 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.047 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.047 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.047 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.048 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.049 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.049 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:32.050 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:32.050 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:32.050 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:32.050 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:32.051 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:32.051 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:32.051 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:32.051 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:32.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.051 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:32.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:32.052 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:32.052 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:32.052 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:32.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:32.053 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:32.053 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.13.12 target=00000000001.10.5 2020-07-17T12:13:37.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:37.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:37.056 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:37.105 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:37.106 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:37.320 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:37.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.321 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:37.321 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.322 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.322 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.322 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.326 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.327 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.328 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.332 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.333 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.334 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.335 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:37.337 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.338 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.339 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.340 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.342 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.343 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.344 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.345 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.346 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.349 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.350 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.351 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.353 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.354 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.355 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.356 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.359 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.360 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:13:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:13:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.362 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.10.5", 2020-07-17T12:13:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:19.855970Z" 2020-07-17T12:13:37.363 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.365 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.367 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.369 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:37.371 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:37.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:37.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.374 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:37.374 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:37.374 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.13.12 target=00000000001.10.5 2020-07-17T12:13:42.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:42.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:42.377 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:42.423 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:42.424 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:42.644 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:42.644 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.645 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:42.645 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.645 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.645 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.645 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.646 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.646 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.646 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.646 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.647 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.647 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.647 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.647 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.648 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:42.648 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.649 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.649 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.649 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.649 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.649 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.649 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.650 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.651 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.651 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.651 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.652 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.652 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.652 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.653 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:42.653 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.653 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.653 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.654 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.654 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.654 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.654 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.655 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.655 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.656 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.656 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.656 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.658 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:42.658 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.658 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.658 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.659 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.659 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.659 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.660 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.666 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.666 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.667 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.667 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.667 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.667 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.668 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.668 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:42.668 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.668 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.668 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.668 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.669 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.669 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.669 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.669 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.670 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.670 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.670 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.671 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.672 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.672 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.672 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:42.673 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.673 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.673 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.673 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.673 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.673 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.674 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.674 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.674 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.675 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.675 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.675 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.675 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.676 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:42.676 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.676 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.676 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.676 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.677 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.677 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.677 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.677 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.678 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.678 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.678 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.679 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.679 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.680 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:42.680 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.680 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.680 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.680 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.681 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.681 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.681 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.684 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.686 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:42.686 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.687 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:42.687 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.687 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.687 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:13:42.687 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.688 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.688 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:13:42.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.688 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.689 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.13.12", 2020-07-17T12:13:42.689 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:13:42.689 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.689 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.690 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:42.690 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.690 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.690 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.690 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.691 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.691 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.691 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.693 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.693 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.693 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.694 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:42.695 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:42.695 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:42.695 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:42.695 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:42.695 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:42.696 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:42.696 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:42.696 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:42.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.696 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:42.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:42.697 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:42.697 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:42.697 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:42.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:42.698 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:42.698 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:42.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:42.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:42.700 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:42.743 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:42.743 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:42.969 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:42.970 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:42.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.970 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:42.971 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.972 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:42.972 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.975 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:42.975 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.976 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:42.976 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.977 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:42.978 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.978 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:42.979 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.980 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:42.980 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.981 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:42.981 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.13.12" 2020-07-17T12:13:42.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.982 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:42.983 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:42.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:42.983 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:42.984 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:42.984 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:42.984 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:42.985 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:42.985 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:42.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:43.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:43.007 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:43.054 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:43.055 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:43.259 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:43.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.260 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:43.260 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.261 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.261 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.261 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.262 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.262 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.262 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.262 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.263 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.263 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.263 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.265 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:43.265 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.265 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.266 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.266 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.266 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.266 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.267 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.267 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.267 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.268 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.268 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.268 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.268 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.269 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:43.269 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.270 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.270 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.270 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.270 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.270 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.272 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.272 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.273 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.273 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.273 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.273 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.274 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:43.275 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.275 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.275 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.275 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.275 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.276 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.276 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.276 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.279 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.279 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.279 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.280 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.280 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:43.280 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.281 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.281 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.281 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.281 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.281 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.282 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.282 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.283 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.283 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.283 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.284 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:43.284 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.284 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.285 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.285 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.285 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.285 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.285 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.286 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.286 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.286 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.287 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.287 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:43.288 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.288 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.288 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.288 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.288 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.289 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.289 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.289 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.289 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.290 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.290 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.291 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:43.291 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.291 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.291 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.292 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.292 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.292 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.292 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.293 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.293 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.293 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.293 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.295 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:43.295 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.295 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:43.296 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.296 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.296 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:13:43.296 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.296 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.297 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:13:43.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.297 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.297 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.13.12", 2020-07-17T12:13:43.297 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:13:43.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.298 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:43.299 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.299 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.299 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.299 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.299 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.300 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.300 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.301 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.302 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.303 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.303 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.305 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:43.305 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:43.311 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:43.311 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:43.312 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:43.312 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:43.313 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:43.313 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:43.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.314 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:43.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.314 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:43.314 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:43.315 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.316 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:43.317 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:43.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:43.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:43.319 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:43.381 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:43.381 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:43.606 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:13:43.609 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:13:43.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.609 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:43.610 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.611 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:43.611 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.612 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:43.612 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.613 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:43.613 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.616 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:43.617 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.617 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:43.617 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.619 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:43.620 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.620 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:43.620 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.621 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:43.621 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.13.12" 2020-07-17T12:13:43.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.622 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:43.622 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:43.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:43.623 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:43.623 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:13:43.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:43.623 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:13:43.623 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:13:43.624 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:43.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:43.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:43.626 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:43.674 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:43.674 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:43.879 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:13:43.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.880 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:43.880 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.880 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.880 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.881 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.881 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.881 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.881 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.882 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.882 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.882 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.883 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.883 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.883 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.884 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:43.884 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.884 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.884 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.884 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.885 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.885 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.885 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.886 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.886 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.886 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.886 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.887 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.887 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.887 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:43.887 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.888 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.888 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.888 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.888 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.888 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.889 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.889 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.890 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.890 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.890 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.890 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.892 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:43.892 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.892 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.892 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.893 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.893 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.893 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.893 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.894 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.895 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.895 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.895 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.895 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.897 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:43.897 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.898 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.898 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.899 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.899 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.899 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.900 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.900 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.901 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.901 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.902 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.902 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.904 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:43.904 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.904 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.905 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.905 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.905 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.906 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.906 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.907 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.907 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.908 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.908 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.909 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.910 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:43.910 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.910 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.911 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.911 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.911 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.912 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.912 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.913 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.913 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.913 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.914 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.915 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:43.915 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.915 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.915 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.915 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.916 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.916 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.916 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.916 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.917 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.917 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.917 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.918 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:43.918 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.919 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:13:43.919 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.919 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.919 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.919 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.920 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.920 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.921 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.921 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.13.12", 2020-07-17T12:13:43.921 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:13:43.921 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.923 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.923 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:43.924 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.924 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.924 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.924 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.924 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.924 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.925 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.925 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.944 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.944 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.944 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.945 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:43.945 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:43.945 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:13:43.946 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:13:43.946 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:13:43.946 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:13:43.946 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:13:43.946 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:13:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:13:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:13:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:13:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:13:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:13:43.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:43.952 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:13:43.953 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:13:43.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:43.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:43.955 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:43.995 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:43.996 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:44.224 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:13:44.224 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:13:44.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.224 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:13:44.225 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.225 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:13:44.225 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.226 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:13:44.226 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.227 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:13:44.227 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.227 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:13:44.228 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.228 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:13:44.228 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.229 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:13:44.229 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.230 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:13:44.230 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.230 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.230 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:13:44.231 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.15.12" 2020-07-17T12:13:44.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.231 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:13:44.231 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:13:44.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:13:44.232 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:13:44.232 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:13:44.232 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:13:44.232 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:13:44.233 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:13:44.234 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:13:44.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:44.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:44.236 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:44.279 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:44.280 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:44.469 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.470 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:44.470 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.470 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.470 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.471 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.471 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.471 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.471 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.472 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.472 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.472 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.472 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.473 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.473 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.473 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:44.474 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.474 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.474 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.474 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.474 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.475 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.475 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.475 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.476 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.476 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.476 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.478 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.479 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:44.480 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.480 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.485 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.485 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.485 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.489 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.492 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.493 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.494 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.495 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:44.496 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.496 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.496 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.497 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.497 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.497 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.497 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.498 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.499 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.499 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.499 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.501 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:44.501 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.502 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.502 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.502 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.505 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.505 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.505 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.507 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:44.507 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.507 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.508 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.508 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.508 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.508 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.509 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.509 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.510 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.510 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.510 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.512 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:44.512 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.513 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:44.513 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.513 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.514 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:13:44.514 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.514 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.515 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:13:44.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.515 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.516 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.11.5", 2020-07-17T12:13:44.516 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:20.142014Z" 2020-07-17T12:13:44.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.517 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:44.518 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.518 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.518 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.518 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.519 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.519 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.519 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.520 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.522 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.522 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.522 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:44.524 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:44.524 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:44.524 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:44.525 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:44.525 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:44.525 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:44.526 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:44.526 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:44.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.526 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:44.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:44.527 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:44.527 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:44.527 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:44.528 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.528 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.528 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:44.529 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:44.529 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.15.12 target=00000000001.11.5 2020-07-17T12:13:49.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:49.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:49.532 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:49.577 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:49.578 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:49.778 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:49.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.779 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:49.779 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.779 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.779 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.779 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.780 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.780 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.780 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.780 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.781 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.781 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.781 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.782 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:49.783 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.783 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.783 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:13:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:13:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.11.5", 2020-07-17T12:13:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:20.142014Z" 2020-07-17T12:13:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:49.835 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:49.835 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.15.12 target=00000000001.11.5 2020-07-17T12:13:54.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:54.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:13:54.838 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:13:54.886 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:13:54.886 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:13:55.089 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:13:55.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.090 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:13:55.090 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.090 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.090 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.091 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.091 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.091 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.091 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.092 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.092 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.092 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.093 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.094 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:13:55.094 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.094 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.095 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.095 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.095 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.095 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.095 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.096 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.097 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.097 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.097 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.098 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:13:55.098 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.099 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.099 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.099 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.099 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.099 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.099 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.100 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.100 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.100 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.101 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.109 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:13:55.110 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.110 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.110 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.111 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.111 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.111 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.111 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.112 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.112 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.112 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.112 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.114 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:13:55.115 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.115 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.115 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.115 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.116 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.116 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.116 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.117 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.117 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.117 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.117 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.118 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:13:55.119 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.119 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.119 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.119 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.119 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.120 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.120 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.120 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.124 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.124 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.124 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.126 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:13:55.126 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.126 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.126 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.127 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.127 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.128 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.128 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.128 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.129 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.129 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.129 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.130 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:13:55.131 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.131 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.131 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.131 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.131 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.132 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.132 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.132 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.133 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.133 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.133 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.134 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:13:55.134 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.134 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:13:55.134 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.135 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.135 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:13:55.135 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.135 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.136 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:13:55.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.136 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.136 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.11.5", 2020-07-17T12:13:55.136 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:20.142014Z" 2020-07-17T12:13:55.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.138 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:13:55.138 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.138 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.138 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.139 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.139 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.139 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.140 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.140 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.142 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.142 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.143 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.143 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.143 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:13:55.144 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:13:55.145 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:13:55.146 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:13:55.146 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:13:55.146 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:13:55.146 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:13:55.147 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:13:55.147 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:13:55.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.147 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:13:55.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:13:55.148 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:13:55.148 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:13:55.149 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:13:55.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:13:55.150 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:13:55.151 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.15.12 target=00000000001.11.5 2020-07-17T12:14:00.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:00.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:00.154 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:00.200 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:00.200 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:00.389 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:00.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.389 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:00.390 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.390 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.390 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.390 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.390 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.393 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:00.393 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.393 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.393 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.399 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.399 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.399 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.399 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.400 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.400 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.400 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.400 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.401 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.401 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.401 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:00.402 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.402 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.402 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.402 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.402 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.402 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.403 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.403 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.403 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.404 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.404 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.405 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:00.405 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.405 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.405 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.405 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.406 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.406 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.406 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.407 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.407 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.407 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.407 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.408 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:00.408 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.409 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.409 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.409 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.409 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.409 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.410 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.410 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.410 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.410 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.411 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.412 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:00.412 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.412 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.412 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.412 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.412 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.413 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.413 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.414 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.414 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.414 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.416 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.417 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:00.417 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.418 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.418 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.418 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.418 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.418 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.419 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.419 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.419 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.419 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.420 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.421 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:00.421 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.421 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.421 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.421 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.421 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.422 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.422 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.422 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.423 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.423 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.423 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.424 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:00.424 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.424 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:00.424 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.425 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.425 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:00.425 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.425 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.425 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:00.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.426 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.426 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.17.9", 2020-07-17T12:14:00.426 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:00.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.427 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.427 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:00.427 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.427 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.428 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.428 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.428 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.428 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.428 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.429 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.430 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.430 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.430 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.430 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.431 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.431 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:00.431 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.432 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.432 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.432 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.432 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.433 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.433 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.433 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.434 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.434 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.434 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.436 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:00.437 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:14:00.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:00.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:00.438 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:00.481 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:00.484 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:00.720 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:14:00.720 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:14:00.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.721 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:00.721 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.721 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.721 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:00.722 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.722 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:00.722 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.723 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:00.723 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.724 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:14:00.724 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.724 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.724 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:14:00.725 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.725 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:14:00.725 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.726 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.726 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:14:00.726 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.726 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.727 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:14:00.727 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000004.17.9" 2020-07-17T12:14:00.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.727 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:14:00.727 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:00.728 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:00.728 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:14:00.728 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:00.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:00.729 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:14:00.729 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:14:00.730 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:14:00.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:00.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:00.732 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:00.775 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:00.776 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:00.982 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:00.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.982 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:00.982 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.983 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.983 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.983 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.983 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.983 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.984 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.984 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.985 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.985 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.985 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.987 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:00.987 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.987 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.987 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.987 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.988 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.988 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.988 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.989 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.989 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.989 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.989 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.990 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:00.991 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.991 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.991 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.991 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.991 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.992 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.992 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.992 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.992 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.993 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.993 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.994 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:00.994 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.994 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.994 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.995 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.995 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:00.995 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:00.996 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:00.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.996 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:00.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.997 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:00.997 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:00.997 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:00.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:00.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:00.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:00.998 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:00.999 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:00.999 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:00.999 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:00.999 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:00.999 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.000 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.000 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.001 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.001 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.002 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.002 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.002 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.002 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.003 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:01.003 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.003 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.003 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.004 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.004 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.004 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.004 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.005 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.005 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.005 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.006 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.014 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:01.014 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.014 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.015 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.015 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.015 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.015 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.015 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.016 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.017 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.017 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.017 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.017 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.017 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.018 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:01.018 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.018 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.019 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.019 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.019 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.019 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.019 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.020 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.020 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.020 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.020 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.021 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:01.022 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.022 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:01.022 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.022 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.022 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:01.023 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.023 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.023 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:01.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.024 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.024 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.17.9", 2020-07-17T12:14:01.026 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:01.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.027 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:01.028 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.028 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.028 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.029 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.029 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.029 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.030 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.030 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.032 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.032 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.033 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.034 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:01.035 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.035 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.035 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.036 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.036 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.036 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.037 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.037 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.038 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.038 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.038 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.039 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:01.040 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:14:01.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:01.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:01.043 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:01.114 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:01.115 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:01.316 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:14:01.316 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:14:01.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.316 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:01.317 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.317 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:01.317 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.318 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:01.318 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.319 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:01.319 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.319 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:01.320 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.320 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:01.321 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.321 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:01.321 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.322 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:01.322 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.323 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:01.323 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.14.9" 2020-07-17T12:14:01.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.324 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:01.324 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.325 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:01.325 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:01.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.325 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:14:01.325 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:14:01.326 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:14:01.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:01.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:01.329 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:01.418 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:01.419 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:01.631 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.12.12", 2020-07-17T12:14:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:01.671 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:01.672 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.14.9 target=00000000003.12.12 2020-07-17T12:14:06.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:06.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:06.674 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:06.721 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:06.721 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:06.917 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:06.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.918 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:06.918 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.918 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.918 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.919 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.919 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.919 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.919 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.919 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.920 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.920 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.920 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.921 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:06.921 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.923 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.923 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.923 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.923 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.923 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.924 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.924 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.924 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.924 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.925 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.926 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:06.926 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.926 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.926 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.926 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.927 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.927 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.927 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.929 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.929 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.929 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.930 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.931 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:06.931 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.931 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.931 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.931 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.932 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.932 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.932 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.932 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.933 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.933 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.933 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.934 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:06.934 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.934 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.935 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.935 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.935 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.935 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.935 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.936 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.936 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.936 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.937 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.938 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:06.938 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.938 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.938 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.938 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.939 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.939 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.939 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.939 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.940 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.940 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.940 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.941 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:06.941 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.941 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.942 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.942 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.942 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.942 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.943 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.943 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.943 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.944 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.944 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.945 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:06.945 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.945 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.945 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.946 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.946 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.946 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.946 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.946 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.947 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.947 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.947 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.948 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.948 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:06.949 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.949 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:06.949 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.949 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.949 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:06.950 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.950 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.950 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:06.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.950 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.951 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.12.12", 2020-07-17T12:14:06.951 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:06.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.952 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:06.952 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.952 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.952 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.952 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.953 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.953 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.953 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.954 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.955 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.955 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.955 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:06.956 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:06.956 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:06.956 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:06.957 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:06.957 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:06.957 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:06.957 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:06.958 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:06.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.958 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:06.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:06.958 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:06.958 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:06.959 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:06.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:06.959 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:06.960 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.14.9 target=00000000003.12.12 2020-07-17T12:14:11.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:11.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:11.963 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:12.007 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:12.007 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:12.206 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:12.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.206 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:12.207 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.207 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.207 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.208 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.208 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.208 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.12.12", 2020-07-17T12:14:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:12.271 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:12.271 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.14.9 target=00000000003.12.12 2020-07-17T12:14:17.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:17.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:17.275 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:17.314 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:17.314 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:17.514 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:17.515 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.516 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.517 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.518 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:17.519 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.520 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.523 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.526 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.527 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:17.530 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.531 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.532 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.534 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:17.537 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.538 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.539 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.541 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.542 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.562 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.563 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.12.12", 2020-07-17T12:14:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.564 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.570 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:17.579 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:17.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:17.579 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:17.580 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.14.9 target=00000000003.12.12 2020-07-17T12:14:22.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:22.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:22.584 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:22.630 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:22.630 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.848 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.850 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.850 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.850 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.851 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.854 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.854 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.854 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.856 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.856 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.856 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.859 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:22.859 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.859 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.859 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.861 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.861 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.861 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.862 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:22.862 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.863 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.863 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.863 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.863 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.868 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.868 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.869 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.870 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.871 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.871 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.873 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:22.874 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.874 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.875 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.875 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.875 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.876 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.876 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.877 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.877 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.878 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.878 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.882 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:22.882 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.882 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.883 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.883 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.883 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.883 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.883 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.884 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.884 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.884 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.885 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.887 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:22.887 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.887 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:22.887 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.887 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.888 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:22.889 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.889 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.889 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:22.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.890 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.890 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.14.9", 2020-07-17T12:14:22.890 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:22.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.891 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:22.891 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.891 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.892 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.892 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.892 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.892 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.892 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.893 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.894 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.894 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.895 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:22.895 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:22.896 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:22.896 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:22.896 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:22.897 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:22.897 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:22.897 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:22.897 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:22.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.898 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:22.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:22.898 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:22.898 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:22.899 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:22.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:22.900 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:14:22.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:22.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:22.902 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:22.973 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:22.974 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:23.194 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:14:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:14:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.195 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:23.196 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.197 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:23.197 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.198 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:23.198 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.198 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:23.198 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.199 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:23.199 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.200 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:23.200 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.200 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:23.201 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.201 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:23.201 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.202 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:23.202 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000004.14.9" 2020-07-17T12:14:23.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.203 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:23.203 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:23.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:23.203 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:23.204 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:23.204 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:23.204 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:14:23.204 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:14:23.205 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:14:23.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:23.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:23.208 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:23.255 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:23.477 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:14:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.478 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.479 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.480 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.482 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.483 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.484 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.499 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.500 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:14:23.501 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.502 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.503 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.503 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.503 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.504 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.504 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.504 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.504 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.505 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:14:23.505 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.505 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.505 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.506 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.506 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.506 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.507 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.507 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.508 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.508 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.509 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.510 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:14:23.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.511 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.511 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.511 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.512 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.512 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.512 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.513 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.514 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.514 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.514 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.515 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.515 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.516 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:14:23.516 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.516 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.517 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.517 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.517 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.518 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.518 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.519 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.522 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.523 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.523 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.523 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.524 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.525 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:14:23.525 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.525 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:14:23.526 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.526 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.526 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:14:23.527 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.527 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.527 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:14:23.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.528 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.528 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000004.14.9", 2020-07-17T12:14:23.528 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:23.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.529 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:14:23.529 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.530 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.530 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.530 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.531 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.531 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.531 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.531 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.533 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.533 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.533 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:23.534 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:14:23.534 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:23.534 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:23.535 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:23.535 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:23.535 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:23.535 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:23.535 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:23.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.536 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:23.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:23.536 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:23.536 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:23.537 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:23.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:23.537 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:14:23.538 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:14:23.538 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-20 zones={test-zone2, test-zone1} 2020-07-17T12:14:23.552 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:14:23.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:23.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:23.608 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:14:23.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:23.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:23.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:23.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:23.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:23.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:23.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:23.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.014 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-20 zones={test-zone2, test-zone1} 2020-07-17T12:14:24.015 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-20 zones={test-zone3, test-zone1} 2020-07-17T12:14:24.034 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:14:24.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:24.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:24.083 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:14:24.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:24.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:24.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.487 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-20 zones={test-zone3, test-zone1} 2020-07-17T12:14:24.488 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-20 zones={test-zone3, test-zone2} 2020-07-17T12:14:24.507 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:14:24.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:24.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:24.575 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:14:24.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:24.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:14:24.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:24.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:14:25.049 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-20 zones={test-zone3, test-zone2} 2020-07-17T12:14:25.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:25.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:25.072 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:25.108 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:25.109 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:25.283 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:14:25.306 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988065.056306_103.1", 2020-07-17T12:14:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:14:25.056306Z" 2020-07-17T12:14:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:14:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:14:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:14:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:14:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:14:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:14:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:25.310 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:25.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:14:25.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:25.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:25.313 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:25.361 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:25.362 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:25.544 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:14:25.545 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:14:25.545 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:14:25.546 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:14:25.546 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:14:25.546 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:14:25.547 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:25.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:25.548 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:14:25.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:25.549 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:25.549 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:25.550 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:14:25.551 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988065.056306_103.1", 2020-07-17T12:14:25.551 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:14:25.551 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:14:25.552 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:14:25.552 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:25.056306Z", 2020-07-17T12:14:25.552 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:25.552 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:25.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:25.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:25.553 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:25.553 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:25.554 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:14:25.554 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:14:25.554 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:14:25.554 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:14:25.554 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:14:25.554 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:14:25.555 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:25.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:25.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:25.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:25.555 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:25.556 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:25.556 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:14:25.556 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:14:25.556 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:14:25.556 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:14:25.557 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:14:25.557 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:14:25.557 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:25.557 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:25.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:25.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:25.559 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:25.559 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:25.559 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:14:25.560 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:14:25.560 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:14:25.560 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:14:25.561 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:14:25.561 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:14:25.561 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:25.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:25.562 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:25.562 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:14:25.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:25.563 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:14:25.563 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:14:25.563 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:14:25.564 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:25.564 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:14:25.565 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:14:25.566 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:14:25.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:25.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:25.568 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:25.613 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:25.614 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:25.782 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:14:25.782 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:14:25.783 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:14:25.783 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:14:25.783 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:14:25.784 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:14:25.784 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:14:25.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:25.784 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:14:25.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:25.785 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:25.785 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:25.785 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:14:25.785 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988065.056306_103.1", 2020-07-17T12:14:25.785 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:14:25.786 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:14:25.786 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:14:25.786 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:14:25.056306Z", 2020-07-17T12:14:25.786 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:14:25.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:25.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:25.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:25.787 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:25.787 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:25.788 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:14:25.788 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:14:25.788 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:14:25.788 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:14:25.788 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:14:25.789 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:14:25.789 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:14:25.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:25.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:25.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:25.790 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:25.790 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:25.790 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:14:25.791 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:14:25.791 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:14:25.791 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:14:25.791 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:14:25.791 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:14:25.792 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:14:25.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:25.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:25.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:25.792 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:25.792 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:25.793 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:14:25.793 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:14:25.793 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:14:25.793 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:14:25.793 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:14:25.794 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:14:25.794 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:14:25.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:25.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:25.795 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:14:25.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:25.795 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:14:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:14:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:14:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:25.797 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:14:25.798 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:14:25.799 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:14:25.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:25.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:25.802 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:25.846 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:25.846 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:26.101 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:14:26.101 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:14:26.101 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:14:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:14:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:14:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:14:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:26.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:14:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.103 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:14:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988065.056306_103.1", 2020-07-17T12:14:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:14:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:14:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:14:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:25.056306Z", 2020-07-17T12:14:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.105 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:26.106 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.107 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:14:26.107 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:14:26.107 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:14:26.107 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:14:26.107 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:14:26.108 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:14:26.108 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:26.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.109 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:26.109 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.109 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:14:26.109 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:14:26.109 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:14:26.110 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:14:26.110 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:14:26.110 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:14:26.110 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:26.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.111 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.111 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:26.111 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.111 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:14:26.111 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:14:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:14:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:14:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:14:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:14:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:14:26.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.113 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.113 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:14:26.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.113 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:14:26.114 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:14:26.114 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:14:26.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.114 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:14:26.116 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:14:26.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:26.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:26.169 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:26.212 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:26.212 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:26.405 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:14:26.405 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:14:26.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.405 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:26.406 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.406 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:26.406 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.407 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:26.407 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.408 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:26.408 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.409 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:26.410 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.411 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:26.411 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.412 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:26.413 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.414 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:26.414 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.415 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:26.416 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000005.17.17" 2020-07-17T12:14:26.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.416 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:26.417 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:26.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:26.418 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:26.418 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:26.419 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:26.419 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:14:26.419 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:14:26.421 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:14:26.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:26.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:26.425 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:26.471 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:26.471 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:26.692 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:14:26.692 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.692 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:26.693 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.693 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.693 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.693 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.693 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.693 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.694 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.694 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.694 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.695 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.695 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.695 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.695 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.696 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:26.696 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.697 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.697 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.697 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.698 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.698 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.698 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.699 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.700 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.700 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.701 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.701 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.701 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.707 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:26.707 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.707 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.708 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.708 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.708 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.708 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.709 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.711 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:14:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.714 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.716 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.718 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:14:26.718 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.718 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.718 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.719 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.719 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.719 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.720 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.720 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.721 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.721 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.722 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.722 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.722 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.723 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:14:26.724 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.724 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.724 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.725 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.726 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.726 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.727 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.727 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.728 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.729 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.729 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:14:26.730 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.730 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.730 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.731 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.731 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.731 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.731 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.732 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.733 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.733 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.733 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.735 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:14:26.735 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.736 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:14:26.736 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.736 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:14:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.738 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:14:26.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.738 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.739 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000005.17.17", 2020-07-17T12:14:26.739 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:14:26.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:14:26.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.740 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.741 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.741 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.741 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.741 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.742 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.743 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.743 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:26.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:14:26.745 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:26.745 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:26.745 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:26.745 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:26.746 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:26.746 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:26.746 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:26.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.746 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:26.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:26.747 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:26.747 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:26.747 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:26.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:26.748 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:14:26.748 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:14:26.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:26.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:26.750 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:26.793 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:26.793 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:27.006 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:14:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:14:27.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.007 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:27.009 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.011 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.012 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:27.013 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000005.17.17" 2020-07-17T12:14:27.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.016 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:14:27.016 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:14:27.017 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:14:27.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:27.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:27.019 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:27.103 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:27.103 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:27.276 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:27.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.277 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.278 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.280 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:27.282 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.283 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.286 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.288 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.289 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.290 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.291 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.292 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:27.294 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.295 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.295 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.296 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.296 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.297 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.297 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.297 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.299 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.300 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.301 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.303 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.305 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.307 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.309 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.310 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.312 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.313 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.315 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:27.318 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.319 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.320 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.322 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.323 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.324 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:27.326 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.327 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.328 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000005.17.17", 2020-07-17T12:14:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:14:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:27.331 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.332 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:27.339 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:27.340 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:27.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.343 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:27.343 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:14:27.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:27.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:27.345 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:27.387 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:27.387 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:27.610 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:14:27.611 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:14:27.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.611 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:27.612 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.612 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:27.613 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.620 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:27.620 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.621 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:27.621 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.622 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:27.623 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.631 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:27.641 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.642 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:27.642 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.643 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:27.643 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.644 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:27.644 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000005.17.17" 2020-07-17T12:14:27.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.646 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:27.646 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:27.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:27.646 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:27.647 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:27.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:27.647 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:14:27.647 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:14:27.648 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:14:27.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:27.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:27.651 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:27.698 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:27.698 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:27.912 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:14:27.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.913 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:27.913 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.913 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.913 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.914 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.914 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.914 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.914 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.915 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.915 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.916 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.917 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.918 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.918 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:27.919 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.919 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.922 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.922 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.922 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.922 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.923 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.923 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.924 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.924 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.924 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.925 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.925 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.926 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:27.926 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.927 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.927 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.927 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.928 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.928 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.929 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.929 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.930 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.930 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.930 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.931 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.939 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.939 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.940 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:27.940 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.940 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.940 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.941 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.941 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.941 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.941 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.942 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.942 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.943 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.943 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.943 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.944 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.944 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:14:27.944 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.945 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.945 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.946 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.946 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.946 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.947 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.947 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.948 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.948 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.948 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.949 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.949 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.950 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:14:27.950 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.950 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.950 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.951 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.951 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.951 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.951 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.952 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.952 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.952 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.953 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.953 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.953 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.954 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:14:27.954 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.955 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.955 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.955 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.955 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.956 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.956 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.957 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.958 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.959 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.959 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.959 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.959 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.960 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.960 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:14:27.960 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.961 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.962 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.962 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.962 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.963 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.963 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.964 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.964 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.964 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.965 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.966 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.966 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:14:27.966 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.966 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:14:27.967 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.967 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.967 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.967 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.968 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.969 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.969 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.970 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000005.17.17", 2020-07-17T12:14:27.970 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:14:27.970 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.970 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.972 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:14:27.972 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.972 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.972 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.973 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.973 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.973 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.974 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.974 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.977 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.978 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.978 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.978 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.979 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:27.979 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:14:27.980 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:27.981 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:27.981 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:27.982 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:27.982 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:27.983 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:27.983 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:27.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.984 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:27.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:27.985 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:27.985 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:27.985 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:27.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.987 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:27.987 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:14:27.988 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-20 2020-07-17T12:14:27.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:27.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:27.990 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:28.031 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:28.032 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:28.253 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:14:28.253 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:14:28.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.255 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:28.255 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.256 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:28.256 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.257 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:28.258 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.259 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:28.259 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.260 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:14:28.261 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.262 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:14:28.262 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.264 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:14:28.264 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.266 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:14:28.266 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.267 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:14:28.267 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000006.22.17" 2020-07-17T12:14:28.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.269 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:14:28.270 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:28.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:28.271 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:14:28.271 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:28.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:28.272 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:14:28.272 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:14:28.274 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:14:28.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:28.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:28.277 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:28.325 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:28.326 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:28.529 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.530 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.531 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.532 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.534 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.537 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.538 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.540 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.541 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.542 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.544 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.545 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.547 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.551 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:28.553 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.554 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.557 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.558 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.560 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.561 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:28.562 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.563 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.564 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.566 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.567 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.568 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.569 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:28.572 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.17.9", 2020-07-17T12:14:28.574 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:28.575 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.576 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.577 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.578 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:28.580 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:28.581 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:28.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:28.584 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:28.584 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.22.17 target=00000000004.17.9 2020-07-17T12:14:33.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:33.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:33.588 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:33.631 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:33.631 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:33.842 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:33.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.843 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:33.843 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.843 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.844 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.844 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.844 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.845 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.845 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.846 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.846 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.847 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.847 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.849 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:33.849 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.850 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.850 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.850 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.851 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.851 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.851 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.852 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.853 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.853 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.853 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.855 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:33.857 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.857 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.857 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.858 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.858 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.858 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.859 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.859 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.860 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.860 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.861 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.862 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:33.863 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.863 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.863 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.864 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.864 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.864 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.865 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.865 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.874 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.875 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.875 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.877 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:33.877 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.877 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.877 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.877 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.877 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.878 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.878 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.879 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.879 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.880 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.880 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.881 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:33.882 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.882 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.882 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.882 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.882 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.883 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.883 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.883 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.884 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.884 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.884 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.885 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:33.885 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.886 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.886 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.886 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.886 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.887 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.887 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.887 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.887 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.888 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.888 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.889 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:33.889 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.889 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.890 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.890 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.890 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.890 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.891 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.891 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.892 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.892 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.892 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.894 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:33.894 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.894 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:33.895 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.895 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.895 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:33.896 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.896 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.897 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:33.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.897 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.898 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.17.9", 2020-07-17T12:14:33.898 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:33.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.899 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:33.900 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.900 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.900 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.900 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.900 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.900 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.901 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.901 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.902 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.903 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.903 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:33.904 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:33.904 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:33.904 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:33.911 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:33.912 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:33.912 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:33.912 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:33.912 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:33.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.913 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:33.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:33.913 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:33.913 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:33.913 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:33.914 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.914 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.914 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:33.914 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:33.914 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.22.17 target=00000000004.17.9 2020-07-17T12:14:38.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:38.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:38.918 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:38.963 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:38.964 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:39.173 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:39.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.174 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:39.174 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.175 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.175 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.175 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.176 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.176 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.177 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.177 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.178 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.178 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.179 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.179 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.179 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.180 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:39.180 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.181 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.182 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.182 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.182 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.183 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.183 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:39.184 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.185 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.185 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.185 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.187 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.187 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.187 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.187 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.187 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.188 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:39.188 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.188 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.189 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.189 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.189 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.189 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.189 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.190 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.190 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.190 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.198 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.199 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:39.199 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.199 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.200 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.200 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.200 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.200 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.201 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.201 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.202 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.202 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.202 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.203 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:39.204 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.204 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.204 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.204 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.205 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.205 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.205 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.206 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.206 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.206 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.206 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.207 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:39.207 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.208 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.208 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.208 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.208 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.208 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.208 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.209 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.209 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.210 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.210 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.212 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:39.212 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.213 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.213 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.213 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.213 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.214 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.214 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.215 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.215 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.216 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.216 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.216 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.216 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.217 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:39.217 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.217 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:39.217 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.218 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.218 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:39.218 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.219 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.219 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:39.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.220 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.220 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.17.9", 2020-07-17T12:14:39.221 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:39.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.222 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.223 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:39.223 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.223 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.224 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.224 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.224 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.225 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.225 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.226 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.227 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.228 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.228 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.228 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:39.230 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:39.230 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:39.230 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:39.231 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:39.231 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:39.231 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:39.232 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:39.232 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:39.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.232 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:39.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:39.233 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:39.233 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:39.234 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:39.234 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.234 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:39.235 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:39.236 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.22.17 target=00000000004.17.9 2020-07-17T12:14:44.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:44.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:44.239 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:44.283 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:44.283 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:44.479 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:44.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.480 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:44.480 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.480 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.481 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.482 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.484 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.485 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.485 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.486 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:44.487 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.488 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.489 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.489 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.489 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.489 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.490 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.490 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.491 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:44.491 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.492 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.492 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.492 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.492 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.493 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.493 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.493 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.494 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.494 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.494 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.495 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:44.496 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.496 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.497 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.497 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.497 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.497 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.498 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.498 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.499 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.499 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.499 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.500 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:44.500 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.500 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.500 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.501 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.501 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.501 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.502 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.502 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:44.504 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.505 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.505 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.505 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.505 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.505 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.506 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.507 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.507 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.507 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.508 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.509 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:44.510 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.510 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.510 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.511 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.511 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.511 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.511 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.512 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.512 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.512 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.512 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.513 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:44.514 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.514 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:44.529 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.529 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.529 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:44.530 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.530 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.530 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:44.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.536 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.536 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000004.17.9", 2020-07-17T12:14:44.537 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:26.122429Z" 2020-07-17T12:14:44.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.538 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:44.538 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.538 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.538 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.538 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.539 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.539 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.539 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.539 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.541 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.541 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.541 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.542 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:44.543 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:44.543 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:44.543 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:44.543 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:44.544 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:44.544 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:44.544 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:44.544 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:44.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.545 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:44.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:44.545 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:44.545 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:44.546 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:44.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.547 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:44.547 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:44.547 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.22.17 target=00000000004.17.9 2020-07-17T12:14:49.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:49.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:49.550 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:49.598 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:49.598 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:49.810 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000007.24.14", 2020-07-17T12:14:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:14:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:49.869 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:49.870 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:14:49.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:49.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:49.872 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:49.914 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:49.915 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:50.150 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:14:50.150 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:14:50.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.151 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:50.151 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.153 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.154 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:50.154 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.155 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:50.155 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.156 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:50.156 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.157 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:14:50.158 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.159 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:14:50.159 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.160 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.160 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:14:50.160 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.161 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:14:50.161 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.162 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:14:50.162 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000007.24.14" 2020-07-17T12:14:50.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.163 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:14:50.163 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:50.164 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:50.164 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:14:50.164 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:14:50.164 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:50.164 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:14:50.164 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:14:50.166 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:14:50.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:50.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:50.167 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:50.214 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:50.214 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:50.413 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:50.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.414 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.417 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.419 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.421 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:50.422 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.423 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.424 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.427 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.428 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.431 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.432 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:50.434 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.435 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.436 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:50.438 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.439 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.440 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.441 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.442 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.443 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.444 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.446 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.449 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.450 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000007.24.14", 2020-07-17T12:14:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:14:50.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.452 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.452 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:50.453 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.453 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.456 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.456 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.456 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.457 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.458 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.460 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.460 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.461 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.461 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.461 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.461 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.462 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:50.463 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:50.463 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:50.463 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:50.464 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:50.464 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:50.465 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:50.465 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:50.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.465 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:50.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.466 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:50.466 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:50.466 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:50.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.467 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:50.467 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-20 2020-07-17T12:14:50.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:50.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:50.469 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:50.543 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:50.544 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:50.763 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:14:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:14:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.772 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:50.773 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.774 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:50.774 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.777 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:50.777 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000007.21.14" 2020-07-17T12:14:50.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.778 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:50.779 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:50.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:14:50.781 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:14:50.783 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:14:50.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:50.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:50.785 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:50.870 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:50.871 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:51.072 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:14:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.073 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.075 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.077 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.080 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.082 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.085 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.089 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.090 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.091 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.092 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.093 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.094 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.096 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.098 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:14:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:14:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:14:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000007.21.14", 2020-07-17T12:14:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:14:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.110 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.112 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:14:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:14:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:14:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:14:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:14:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:14:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:14:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:14:51.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:14:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:14:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.116 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.116 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:14:51.116 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:14:51.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:51.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:51.118 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:51.197 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:51.197 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:51.402 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:14:51.403 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:14:51.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.403 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:14:51.403 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.404 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:14:51.404 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.404 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:14:51.405 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.406 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:14:51.406 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.406 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:14:51.407 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.408 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:14:51.408 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.409 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:14:51.409 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.410 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:14:51.410 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.418 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:14:51.419 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000007.21.14" 2020-07-17T12:14:51.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.419 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:14:51.420 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:14:51.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:14:51.421 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:14:51.421 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:14:51.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:14:51.429 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:14:51.430 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:14:51.431 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:14:51.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:51.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:51.435 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:51.483 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:51.483 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:51.696 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:14:51.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.697 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:51.697 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.697 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.697 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.698 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.698 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.698 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.698 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.699 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.699 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.699 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.699 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.700 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:51.700 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.701 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.701 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.701 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.701 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.701 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.706 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.707 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.707 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.708 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.708 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.708 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.708 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.709 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:51.709 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.709 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.709 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.709 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.710 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.710 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.710 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.711 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.713 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:51.713 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.713 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.713 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.726 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.728 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.729 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.734 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.734 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.735 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.735 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.736 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.736 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.736 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.737 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:14:51.737 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.737 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.738 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.738 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.739 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.739 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.739 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.740 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.740 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.740 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.741 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.744 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:14:51.744 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.744 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.745 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.746 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.747 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.747 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.747 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.748 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.748 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.749 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.749 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.749 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.749 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.750 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:14:51.750 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.750 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.750 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.751 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.751 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.751 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.751 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.752 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.752 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.752 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.752 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.752 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.753 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:14:51.754 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.754 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.755 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.755 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.755 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.755 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.755 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.756 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.756 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.756 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.756 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.757 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:14:51.757 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.758 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:14:51.758 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.758 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.758 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:14:51.758 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.759 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.759 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:14:51.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.759 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.760 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.19.17", 2020-07-17T12:14:51.760 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:14:51.760 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.760 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.761 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:14:51.761 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.761 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.761 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.761 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.762 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.762 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.762 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.764 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.764 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.764 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.765 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.765 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:51.767 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:14:51.767 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:51.767 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:51.767 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:51.767 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:51.767 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:51.768 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:51.768 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:51.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.768 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:51.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:51.769 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:51.769 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:51.769 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:51.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:51.770 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:14:51.770 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone2 behind zone test-zone3: shard=8 master=00000000007.21.14 target=00000000006.19.17 2020-07-17T12:14:56.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:56.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:14:56.773 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:14:56.826 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:14:56.826 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:14:57.046 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:14:57.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.046 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:14:57.047 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.047 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.047 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.047 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.047 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.048 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.048 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.048 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.049 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.049 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.049 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.051 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:14:57.051 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.051 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.051 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.051 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.052 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.052 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.058 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.059 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.059 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.059 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.060 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.060 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.060 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.061 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:14:57.061 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.061 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.061 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.062 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.062 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.062 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.062 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.064 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.065 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.065 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.065 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.067 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:14:57.067 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.068 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.068 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.068 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.068 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.068 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.069 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.069 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.070 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.070 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.070 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.071 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:14:57.072 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.072 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.072 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.072 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.072 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.073 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.073 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.074 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.074 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.074 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.075 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.077 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:14:57.077 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.079 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.079 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.080 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.080 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.081 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.081 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.082 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.082 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.082 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.083 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.084 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.085 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:14:57.085 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.085 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.085 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.085 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.086 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.086 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.086 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.087 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.087 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.088 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.088 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.089 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:14:57.089 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.089 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.090 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.090 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.090 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.090 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.090 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.091 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.091 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.091 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.092 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.092 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.092 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.093 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:14:57.093 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.093 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:14:57.093 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.093 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.094 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:14:57.094 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.095 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.095 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:14:57.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.095 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.096 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.19.17", 2020-07-17T12:14:57.096 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:14:57.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.097 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:14:57.097 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.098 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.098 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.098 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.098 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.098 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.099 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.100 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.100 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.101 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.101 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:14:57.102 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:14:57.102 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:14:57.102 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:14:57.102 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:14:57.103 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:14:57.103 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:14:57.103 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:14:57.103 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:14:57.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.104 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:14:57.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:14:57.104 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:14:57.104 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:14:57.105 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:14:57.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.107 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:14:57.107 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:14:57.107 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone2 behind zone test-zone3: shard=8 master=00000000007.21.14 target=00000000006.19.17 2020-07-17T12:15:02.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:02.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:02.113 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:02.157 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:02.157 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:02.365 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:02.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.365 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:02.366 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.366 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.366 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.366 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.367 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.367 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.367 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.368 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.369 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.369 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.369 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.369 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.371 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:02.371 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.371 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.371 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.372 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.372 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.372 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.372 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.373 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.373 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.373 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.373 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.374 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.374 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.374 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.374 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:02.375 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.375 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.375 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.375 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.376 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.376 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.376 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.376 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.377 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.377 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.378 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.378 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.378 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.379 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:02.379 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.380 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.380 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.380 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.381 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.381 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.382 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.383 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.384 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.385 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.385 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.385 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.385 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.386 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.386 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:02.387 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.387 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.387 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.388 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.388 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.389 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.389 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.389 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.390 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.390 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.390 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.390 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.391 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.391 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.391 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.391 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:02.392 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.392 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.393 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.393 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.393 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.394 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.394 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.395 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.396 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.396 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.396 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.397 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.397 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.397 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.398 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.398 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:02.398 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.399 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.399 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.399 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.399 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.400 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.400 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.401 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.402 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.408 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.408 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.410 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:02.410 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.411 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.411 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.411 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.411 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.412 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.412 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.413 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.413 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.413 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.419 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.420 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.420 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.420 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.420 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:02.421 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.421 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:02.421 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.421 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.421 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:02.422 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.422 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.422 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:02.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.422 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.423 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000006.19.17", 2020-07-17T12:15:02.423 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:15:02.423 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.423 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.424 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:02.424 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.424 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.424 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.424 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.425 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.425 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.425 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.426 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.427 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.427 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.427 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.427 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.427 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.428 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:02.428 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:02.428 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:02.428 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:02.429 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:02.430 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:02.430 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:02.430 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:02.430 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:02.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.431 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:02.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:02.432 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:02.432 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:02.432 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:02.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:02.433 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:02.434 WARNING:rgw_multi.tests:bucket lbjidp-20 zone test-zone2 behind zone test-zone3: shard=8 master=00000000007.21.14 target=00000000006.19.17 2020-07-17T12:15:07.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:07.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:07.436 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:07.482 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:07.482 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:07.702 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:07.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.703 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:07.703 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.703 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.703 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.704 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.704 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.704 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.704 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.705 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.705 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.705 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.705 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.706 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:07.706 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.707 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.707 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.707 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.707 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.707 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.708 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.708 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.708 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.708 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.709 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.710 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:07.710 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.710 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.710 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.710 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.710 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.711 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.711 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.711 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.712 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.712 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.715 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.715 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.715 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.716 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:07.717 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.717 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.717 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.717 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.717 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.718 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.718 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.718 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.719 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.719 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.719 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.719 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.720 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.721 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:07.721 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.721 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.721 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.721 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.722 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.722 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.722 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.723 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.723 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.724 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.724 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.725 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:07.725 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.725 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.725 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.726 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.726 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.726 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.726 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.726 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.727 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.728 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.728 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.729 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.729 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:07.729 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.729 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.729 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.730 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.730 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.730 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.730 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.731 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.731 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.731 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.731 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.732 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.732 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.732 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:07.733 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.733 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.734 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.734 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.734 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.734 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.734 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.735 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.735 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.735 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.735 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.736 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.736 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.736 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.736 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:07.737 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.737 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:07.737 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.737 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.738 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:07.738 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.738 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.738 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:07.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.739 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.739 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000007.21.14", 2020-07-17T12:15:07.739 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:26.118419Z" 2020-07-17T12:15:07.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:07.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.740 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.740 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.741 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.741 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.741 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.741 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.742 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.743 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.743 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.743 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.747 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:07.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:07.747 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:07.747 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:07.747 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:07.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:07.748 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:07.748 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-20 2020-07-17T12:15:07.749 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-20 zones={test-zone2, test-zone1} 2020-07-17T12:15:07.763 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:07.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:07.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:07.816 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:07.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:07.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:07.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:07.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:07.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.175 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-20 zones={test-zone2, test-zone1} 2020-07-17T12:15:08.176 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-20 zones={test-zone3, test-zone1} 2020-07-17T12:15:08.186 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:08.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:08.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:08.236 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:08.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:08.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:08.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.630 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-20 zones={test-zone3, test-zone1} 2020-07-17T12:15:08.636 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-20 zones={test-zone3, test-zone2} 2020-07-17T12:15:08.650 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:08.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:08.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:08.700 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:08.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:08.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:08.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:08.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:09.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:09.086 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-20 zones={test-zone3, test-zone2} 2020-07-17T12:15:09.093 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-07-17T12:15:09.095 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-21 2020-07-17T12:15:09.126 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-22 2020-07-17T12:15:09.182 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-23 2020-07-17T12:15:09.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:09.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:09.279 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:09.315 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:09.315 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:09.485 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:15:09.505 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:09.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:09.506 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988109.262832_133.1", 2020-07-17T12:15:09.506 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:15:09.262832Z" 2020-07-17T12:15:09.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:09.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:09.507 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:09.507 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:15:09.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:09.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:09.508 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:09.508 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:15:09.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:09.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:09.509 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:09.509 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:15:09.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:09.510 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:09.511 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:15:09.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:09.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:09.513 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:09.573 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:09.573 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:09.812 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:09.813 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:15:09.813 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:15:09.813 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:15:09.814 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:15:09.814 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:09.814 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:09.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:09.814 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:09.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:09.815 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:09.815 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:09.815 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:09.815 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988109.262832_133.1", 2020-07-17T12:15:09.816 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:09.816 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:15:09.816 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:09.816 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.262832Z", 2020-07-17T12:15:09.816 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:09.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:09.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:09.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:09.817 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:09.817 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:09.817 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:09.818 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:09.818 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:09.818 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:09.818 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:09.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:09.819 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:09.819 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:09.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:09.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:09.819 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:09.819 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:09.820 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:09.820 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:09.820 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:09.820 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:09.820 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:09.821 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:09.821 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:09.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:09.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:09.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:09.821 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:09.831 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:09.832 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:09.832 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:09.832 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:09.833 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:09.833 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:09.833 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:09.834 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:09.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:09.835 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:09.835 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:09.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:09.836 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:15:09.836 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:15:09.836 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:15:09.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:09.838 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:09.840 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:15:09.841 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:15:09.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:09.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:09.843 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:09.884 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:09.884 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:10.095 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:15:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:15:10.096 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:15:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:15:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:10.097 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.098 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:10.099 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988109.262832_133.1", 2020-07-17T12:15:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:10.100 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:15:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.262832Z", 2020-07-17T12:15:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:10.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:10.102 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:10.103 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:10.114 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:10.115 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:15:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:10.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:10.118 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:10.119 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:15:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:10.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:15:10.125 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:15:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:15:10.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:10.126 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:10.137 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:15:10.138 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:15:10.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:10.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:10.139 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:10.191 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:10.192 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:10.468 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:10.468 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:15:10.469 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:15:10.469 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:15:10.470 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:15:10.470 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:10.470 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:10.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:10.471 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:10.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:10.471 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:10.472 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:10.472 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:10.472 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988109.262832_133.1", 2020-07-17T12:15:10.473 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:10.473 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:15:10.473 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:10.479 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.262832Z", 2020-07-17T12:15:10.480 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:10.480 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:10.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:10.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:10.482 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:10.482 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:10.482 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:10.483 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:10.483 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:10.483 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:10.483 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:10.484 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:10.484 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:10.484 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:10.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:10.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:10.485 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:10.486 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:10.486 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:10.487 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:10.487 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:10.487 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:10.487 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:10.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:10.488 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:10.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:10.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:10.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:10.489 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:10.490 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:10.490 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:10.490 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:10.491 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:10.491 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:10.491 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:10.492 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:10.492 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:10.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:10.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:10.493 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:10.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:10.494 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:15:10.494 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:15:10.494 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:15:10.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:10.495 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:10.496 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:15:10.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:10.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:10.723 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:10.766 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:10.766 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:10.945 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.950 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:10.950 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.14.11" 2020-07-17T12:15:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:10.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:10.956 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:10.956 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:10.957 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-21 2020-07-17T12:15:10.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:10.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:10.958 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:11.003 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:11.003 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:11.182 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:11.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.183 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:11.183 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.183 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.184 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.184 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.184 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.184 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.185 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.185 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.185 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.188 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:11.188 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.191 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.191 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.197 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.197 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:11.198 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.199 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.200 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.201 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.202 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:11.203 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.203 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.203 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.203 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.204 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.206 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:11.206 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.207 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.207 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.207 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.208 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.208 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.208 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.209 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.210 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.211 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.211 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.211 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.213 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:11.213 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.213 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.214 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.214 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.214 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.216 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.216 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.218 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.219 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.219 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.220 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.221 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:11.222 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.222 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.223 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.223 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.223 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.224 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.224 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.226 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.227 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.227 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.227 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.228 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.228 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.229 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:11.229 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.230 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.230 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.231 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.231 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.231 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.232 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.232 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.233 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.234 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.234 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.235 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.236 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:11.236 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.237 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:11.238 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.238 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.238 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:11.239 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "KpYMTmLR5q-LtWjfhTxTweF6STYjwCj", 2020-07-17T12:15:11.239 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.239 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:11.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.239 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.240 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.14.11", 2020-07-17T12:15:11.240 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:10.715935Z" 2020-07-17T12:15:11.240 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.240 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.241 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:11.241 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.241 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.241 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.241 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.242 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.242 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.242 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.243 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.244 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.244 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.244 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:11.245 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:11.245 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:11.245 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:11.245 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:11.246 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:11.246 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:11.246 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:11.246 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:11.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.247 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:11.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:11.247 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:11.247 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:11.247 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.247 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.248 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.248 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:11.248 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:11.248 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-21 2020-07-17T12:15:11.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:11.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:11.250 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:11.290 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:11.290 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:11.497 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:11.498 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:11.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.498 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:11.498 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.499 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:11.499 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.500 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:11.500 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.501 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:11.501 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.501 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:11.501 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.502 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:11.502 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.503 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:11.503 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.503 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:11.504 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.504 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:11.504 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.14.11" 2020-07-17T12:15:11.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.505 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:11.506 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.515 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:11.516 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:11.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:11.517 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:11.519 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-21 2020-07-17T12:15:11.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:11.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:11.521 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:11.607 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:11.607 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:11.817 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:11.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.818 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:11.819 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.819 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.819 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.819 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.820 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.820 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.820 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.821 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.821 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.821 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.822 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.823 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:11.823 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.824 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.824 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.824 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.827 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.828 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.828 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.828 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.829 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.829 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.829 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.829 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.830 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:11.831 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.831 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.831 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.832 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.832 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.832 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.833 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.833 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.848 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.848 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.849 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.850 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:11.850 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.851 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.851 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.851 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.852 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.852 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.852 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.853 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.860 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.860 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.862 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.863 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:11.863 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.864 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.867 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.868 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.868 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.870 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.870 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.872 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.872 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.872 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.873 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.874 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.875 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.877 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.878 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.878 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.878 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.879 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.879 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.880 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.882 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:11.882 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.882 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.885 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:11.886 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.886 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:11.886 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.886 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.886 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:11.886 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KpYMTmLR5q-LtWjfhTxTweF6STYjwCj", 2020-07-17T12:15:11.887 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.887 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:11.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.887 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.888 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.14.11", 2020-07-17T12:15:11.888 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:10.713945Z" 2020-07-17T12:15:11.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.889 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:11.889 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.889 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.889 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.889 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.890 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.890 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.890 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.892 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.892 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.892 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.892 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:11.893 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:11.894 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:11.894 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:11.894 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:11.894 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:11.894 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:11.894 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:11.895 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:11.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.895 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:11.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:11.895 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:11.896 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:11.896 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:11.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:11.896 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:11.897 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-21 2020-07-17T12:15:11.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:11.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:11.899 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:11.945 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:11.945 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:12.169 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:12.170 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:12.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.170 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:12.170 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.171 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:12.171 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.172 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:12.172 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.173 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:12.173 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.174 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:12.174 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.175 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:12.175 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.176 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:12.177 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.177 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:12.178 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.178 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:12.179 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.14.11" 2020-07-17T12:15:12.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.180 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:12.180 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:12.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:12.181 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:12.181 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:12.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:12.181 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:12.181 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:12.182 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-21 2020-07-17T12:15:12.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:12.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:12.184 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:12.229 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:12.229 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:12.449 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:12.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.450 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:12.450 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.450 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.450 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.450 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.451 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.451 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.451 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.451 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.452 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.452 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.452 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.453 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:12.453 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.453 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.454 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.455 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.455 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.455 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.455 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.456 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.456 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.456 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.456 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.457 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.457 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.457 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:12.459 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.459 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.459 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.459 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.460 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.460 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.460 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.460 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.462 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:12.462 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.462 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.463 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.463 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.463 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.463 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.463 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.464 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.466 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.466 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.466 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.466 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.467 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.467 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:12.467 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.468 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.468 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.468 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.468 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.468 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.469 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.469 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.469 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.471 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:12.471 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.471 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.471 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.471 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.472 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.472 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.472 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.472 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.473 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.473 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.473 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.473 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.473 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.474 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:12.474 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.474 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.475 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.475 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.476 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.476 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.476 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.476 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.477 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.477 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.477 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.477 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.478 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.479 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:12.479 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.479 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.480 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.480 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.480 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.480 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.480 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.481 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.481 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.481 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.481 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.482 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.483 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.484 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:12.484 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.484 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:12.484 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.484 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.485 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.486 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.486 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.486 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.487 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.487 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.14.11", 2020-07-17T12:15:12.487 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:10.549553Z" 2020-07-17T12:15:12.487 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.487 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.488 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:12.488 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.488 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.489 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.489 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.489 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.489 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.489 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.490 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.504 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.507 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:12.507 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:12.507 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:12.508 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:12.508 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:12.508 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:12.509 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:12.509 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:12.511 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:12.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.512 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:12.513 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-21 2020-07-17T12:15:12.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:12.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:12.514 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:12.559 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:12.560 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:12.790 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:12.791 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:12.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.791 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:12.791 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.792 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:12.792 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.793 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:12.793 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.793 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:12.794 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.794 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.794 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:12.795 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.795 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.795 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:12.795 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.796 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:12.796 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.797 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:12.797 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.797 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:12.798 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000002.16.5" 2020-07-17T12:15:12.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.798 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.798 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:12.798 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:12.799 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:12.799 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:12.799 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:12.799 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:12.799 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:12.800 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:12.801 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-21 2020-07-17T12:15:12.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:12.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:12.811 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:12.855 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:12.856 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:13.069 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KpYMTmLR5q-LtWjfhTxTweF6STYjwCj", 2020-07-17T12:15:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.16.5", 2020-07-17T12:15:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:11.049997Z" 2020-07-17T12:15:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.113 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:13.114 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-21 2020-07-17T12:15:13.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:13.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:13.115 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:13.154 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:13.154 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:13.399 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:13.530 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000002.16.5" 2020-07-17T12:15:13.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.531 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:13.531 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:13.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:13.532 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:13.532 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:13.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:13.533 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:13.534 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:13.535 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-21 2020-07-17T12:15:13.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:13.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:13.538 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:13.593 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:13.594 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:13.804 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.805 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.807 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:13.809 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.815 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.817 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.818 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.820 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.821 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.822 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.823 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.823 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.824 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.824 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.824 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.826 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:13.826 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.826 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.827 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.827 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.827 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.827 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.827 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.828 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.828 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.828 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.828 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.829 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.829 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.829 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:13.830 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.830 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.830 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.831 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.831 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.831 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.832 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.832 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.832 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.833 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.833 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.835 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:13.835 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.835 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.836 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.836 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.836 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.837 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.837 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.838 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.838 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.838 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.838 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.839 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:13.840 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.840 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.840 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.840 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.840 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.841 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.841 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.841 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.842 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.842 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.842 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.843 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.843 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:13.844 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.844 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.844 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.844 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.844 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.845 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.845 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.845 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.846 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.846 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.846 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.848 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:13.848 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.848 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:13.849 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.849 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.849 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:13.850 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KpYMTmLR5q-LtWjfhTxTweF6STYjwCj", 2020-07-17T12:15:13.850 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.851 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:13.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.851 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.852 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.16.5", 2020-07-17T12:15:13.852 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:11.007401Z" 2020-07-17T12:15:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.854 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:13.854 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.855 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.855 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.855 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.856 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.856 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.856 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.858 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.859 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.859 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.860 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:13.861 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:13.862 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:13.862 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:13.862 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:13.863 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:13.863 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:13.863 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:13.864 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:13.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.865 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:13.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:13.865 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:13.866 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:13.866 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:13.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:13.867 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:13.868 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-21 2020-07-17T12:15:13.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:13.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:13.871 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:13.940 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:13.940 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:14.155 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:14.155 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:14.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.157 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:14.157 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.158 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:14.158 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.159 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:14.159 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.160 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:14.160 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.161 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:14.161 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.162 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:14.162 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.181 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:14.181 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.182 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:14.183 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.185 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:14.185 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.17.5" 2020-07-17T12:15:14.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.190 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:14.190 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.191 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:14.191 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.191 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:14.192 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:14.193 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-21 2020-07-17T12:15:14.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:14.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:14.195 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:14.278 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:14.279 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:14.487 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:14.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.488 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.489 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.490 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:14.494 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.495 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.496 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.497 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.510 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.511 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.512 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.514 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.515 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:14.519 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.520 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.521 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.522 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.524 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.527 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.528 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.530 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.533 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.534 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.534 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.534 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.534 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.535 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.535 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.536 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.536 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.536 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.538 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:14.538 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.539 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.539 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.539 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.540 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.540 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.540 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.541 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.542 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.542 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.546 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.547 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:14.547 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.547 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.548 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.548 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.548 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.548 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.548 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.549 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.549 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.549 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.550 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.551 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:14.551 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.551 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:14.551 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.552 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.553 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:14.554 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "KpYMTmLR5q-LtWjfhTxTweF6STYjwCj", 2020-07-17T12:15:14.554 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.554 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:14.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.555 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.555 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.17.5", 2020-07-17T12:15:14.555 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:10.501422Z" 2020-07-17T12:15:14.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.556 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:14.557 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.557 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.557 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.557 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.558 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.558 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.558 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.560 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.560 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.560 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.561 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.561 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.561 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.562 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:14.562 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:14.562 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:14.562 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:14.563 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:14.563 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:14.563 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:14.563 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:14.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.564 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:14.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.564 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:14.564 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:14.564 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:14.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.565 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:14.566 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-21 2020-07-17T12:15:14.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:14.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:14.568 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:14.647 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:14.648 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:14.852 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.853 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.855 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:14.856 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.858 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:14.859 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.861 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:14.862 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.862 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:14.863 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.863 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:14.863 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.17.5" 2020-07-17T12:15:14.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.864 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:14.864 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:14.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:14.865 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:14.866 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:14.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:14.866 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:14.866 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:14.867 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-21 2020-07-17T12:15:14.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:14.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:14.869 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:14.915 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:14.915 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:15.122 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:15.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.123 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:15.123 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.125 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.125 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.125 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.126 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.126 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.126 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.127 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.128 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.128 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.129 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.129 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.130 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:15.131 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.131 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.131 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.132 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.132 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.132 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.133 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.133 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.133 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.134 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.134 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.135 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:15.135 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.135 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.135 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.136 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.136 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.136 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.136 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.137 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.137 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.138 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.138 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.139 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:15.139 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.139 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.140 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.140 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.140 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.140 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.140 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.141 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.141 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.141 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.141 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.142 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:15.143 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.143 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.143 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.143 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.143 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.143 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.144 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.144 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.144 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.145 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.145 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.146 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:15.146 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.146 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.146 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.146 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.147 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.147 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.147 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.147 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.148 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.148 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.148 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.149 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:15.149 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.149 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.149 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.150 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.150 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.150 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.150 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.151 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.151 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.151 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.151 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.152 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:15.152 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.152 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.153 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.153 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.153 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.153 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.154 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.154 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.155 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.155 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.156 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.157 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.157 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:15.158 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.158 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:15.158 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.159 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.159 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:15.159 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "KpYMTmLR5q-LtWjfhTxTweF6STYjwCj", 2020-07-17T12:15:15.159 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.161 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:15.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.161 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.162 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.17.5", 2020-07-17T12:15:15.163 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:10.501422Z" 2020-07-17T12:15:15.163 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.165 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.166 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:15.166 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.167 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.167 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.167 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.167 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.167 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.168 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.169 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.169 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.170 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.170 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:15.172 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:15.172 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:15.172 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:15.172 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:15.173 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:15.173 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:15.173 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:15.173 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:15.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.174 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:15.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:15.174 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:15.174 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:15.174 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:15.174 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:15.175 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:15.175 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-21 2020-07-17T12:15:15.176 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-21 zones={test-zone2, test-zone1} 2020-07-17T12:15:15.188 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:15.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.239 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:15.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:15.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:15.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:15.399 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-21 zones={test-zone2, test-zone1} 2020-07-17T12:15:15.400 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-21 zones={test-zone3, test-zone1} 2020-07-17T12:15:15.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:15.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.460 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:15.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:15.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:15.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:15.619 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-21 zones={test-zone3, test-zone1} 2020-07-17T12:15:15.619 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-21 zones={test-zone3, test-zone2} 2020-07-17T12:15:15.629 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:15.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.676 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:15.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:15.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:15.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:15.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:15.842 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-21 zones={test-zone3, test-zone2} 2020-07-17T12:15:15.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:15.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:15.851 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:15.898 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:15.898 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:16.127 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:16.127 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:16.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.127 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:16.128 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.128 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:16.129 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.130 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:16.130 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.131 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:16.131 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.131 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:16.132 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.132 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:16.132 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.133 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:16.133 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.135 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:16.135 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.135 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:16.136 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.9.5" 2020-07-17T12:15:16.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.136 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:16.137 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.137 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:16.137 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:16.138 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:16.138 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:16.139 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-22 2020-07-17T12:15:16.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:16.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:16.140 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:16.189 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:16.190 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:16.413 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:16.414 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.414 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:16.414 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.414 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.415 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.415 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.415 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.415 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.415 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.416 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.416 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.416 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.417 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.417 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.417 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.418 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:16.418 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.418 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.418 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.419 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.419 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.419 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.419 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.420 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.420 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.420 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.420 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.420 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.421 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.421 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:16.421 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.422 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.422 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.422 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.422 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.422 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.423 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.423 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.423 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.423 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.424 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.424 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.424 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.425 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:16.425 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.425 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.425 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.425 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.425 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.428 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.429 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.429 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.429 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.429 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.430 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.431 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:16.431 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.431 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.431 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.431 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.432 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.432 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.432 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.432 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.433 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.433 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.433 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.434 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.434 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:16.434 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.434 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.435 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.435 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.435 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.435 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.435 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.436 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.436 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.436 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.436 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.437 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.437 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.437 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.437 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:16.438 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.438 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.438 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.438 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.438 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.438 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.439 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.439 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.439 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.440 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.440 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.440 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.441 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:16.441 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.441 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.441 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.441 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.442 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.442 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.442 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.442 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.443 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.443 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.443 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.444 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:16.444 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.444 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:16.444 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.445 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.445 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:16.445 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jmNxEOgz4yiLDvZpPoUGl3lbqknNMpL", 2020-07-17T12:15:16.445 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.446 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 2 2020-07-17T12:15:16.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.446 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.446 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.9.5", 2020-07-17T12:15:16.446 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:10.955821Z" 2020-07-17T12:15:16.447 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.447 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.447 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:16.448 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.448 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.448 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.448 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.449 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.449 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.449 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.451 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.451 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.451 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.451 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:16.452 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:16.452 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:16.453 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:16.453 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:16.453 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:16.453 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:16.453 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:16.454 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:16.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.454 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:16.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:16.454 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:16.454 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:16.455 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:16.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:16.455 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:16.461 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-22 2020-07-17T12:15:16.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:16.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:16.463 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:16.506 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:16.506 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:16.724 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:16.725 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:16.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.725 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:16.726 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.727 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:16.727 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.728 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:16.728 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.728 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:16.729 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.730 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:16.730 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.731 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:16.731 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.731 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:16.732 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.732 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:16.732 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.733 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:16.734 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.9.5" 2020-07-17T12:15:16.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.734 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:16.735 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:16.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:16.735 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:16.736 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:16.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:16.736 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:16.737 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:16.738 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-22 2020-07-17T12:15:16.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:16.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:16.741 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:16.824 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:16.825 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:17.031 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:17.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.032 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:17.032 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.032 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.032 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.033 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.033 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.033 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.033 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.034 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.034 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.034 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.034 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.035 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:17.035 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.036 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.036 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.036 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.036 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.036 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.037 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.037 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.037 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.038 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.038 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.039 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:17.039 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.040 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.040 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.040 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.040 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.040 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.041 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.041 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.041 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.048 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.048 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.049 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:17.050 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.050 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.050 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.050 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.050 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.050 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.051 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.051 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.051 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.052 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.052 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.053 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:17.053 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.053 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.053 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.053 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.054 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.054 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.054 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.054 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.055 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.055 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.055 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.056 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:17.056 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.056 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.057 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.057 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.057 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.057 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.057 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.063 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.064 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.064 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.064 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.066 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:17.066 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.066 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.067 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.067 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.067 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.068 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.068 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.068 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.069 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.069 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.070 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.079 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:17.080 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.080 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.081 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.081 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.081 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.082 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.082 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.082 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jmNxEOgz4yiLDvZpPoUGl3lbqknNMpL", 2020-07-17T12:15:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 2 2020-07-17T12:15:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.9.5", 2020-07-17T12:15:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:10.961919Z" 2020-07-17T12:15:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.100 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:17.100 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-22 2020-07-17T12:15:17.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:17.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:17.103 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:17.184 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:17.185 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:17.402 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:17.403 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:17.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.403 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:17.403 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.404 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:17.404 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.404 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:17.405 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.405 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:17.405 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.406 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:17.406 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.407 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:17.407 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.407 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:17.408 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.408 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:17.408 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.409 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:17.409 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.9.5" 2020-07-17T12:15:17.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.410 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:17.410 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:17.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:17.410 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:17.410 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:17.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:17.411 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:17.411 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:17.412 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-22 2020-07-17T12:15:17.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:17.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:17.414 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:17.459 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:17.459 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:17.678 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:17.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.679 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:17.679 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.679 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.679 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.679 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.680 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.680 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.680 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.680 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.681 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.681 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.681 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.682 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.682 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:17.683 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.683 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.683 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.683 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.683 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.683 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.684 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.684 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.684 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.685 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.685 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.686 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:17.686 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.686 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.686 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.686 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.687 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.687 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.690 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.690 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.691 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.691 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.691 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.692 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.692 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:17.692 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.692 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.693 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.693 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.693 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.693 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.693 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.694 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.694 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.694 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.695 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.695 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.695 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.695 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:17.696 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.696 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.696 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.696 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.696 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.697 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.697 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.697 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.698 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.698 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.699 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.700 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:17.700 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.700 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.700 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.700 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.701 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.701 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.701 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.701 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.702 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.702 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.702 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.703 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.703 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.703 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:17.704 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.704 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.704 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.704 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.704 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.704 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.705 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.705 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.705 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.706 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.706 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.707 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:17.707 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.707 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.707 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.707 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.708 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.708 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.708 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.709 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.709 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.709 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.709 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.710 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:17.710 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.711 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:17.711 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.711 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.711 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.711 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.712 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.712 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.712 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.712 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.9.5", 2020-07-17T12:15:17.713 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:10.573032Z" 2020-07-17T12:15:17.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.714 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:17.714 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.714 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.714 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.714 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.715 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.715 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.715 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.715 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.733 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.733 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.734 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:17.735 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:17.736 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:17.736 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:17.736 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:17.736 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:17.736 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:17.736 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:17.737 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:17.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.737 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:17.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:17.738 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:17.738 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:17.738 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:17.738 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.738 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:17.739 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:17.740 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-22 2020-07-17T12:15:17.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:17.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:17.742 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:17.822 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:17.822 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:18.048 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:18.049 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:18.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.049 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:18.049 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.050 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:18.050 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.051 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:18.051 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.052 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:18.052 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.053 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:18.059 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.059 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:18.059 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.061 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:18.061 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.070 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:18.070 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.14.11" 2020-07-17T12:15:18.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.071 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:18.071 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.072 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:18.072 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:18.072 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:18.072 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:18.073 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-22 2020-07-17T12:15:18.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:18.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:18.076 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:18.119 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:18.120 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:18.332 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:18.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.333 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:18.333 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.334 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.334 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.334 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.335 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.335 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.336 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.336 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.337 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.338 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.338 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.340 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:18.341 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.341 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.341 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.342 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.342 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.343 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.343 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.344 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.344 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.345 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.345 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.347 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:18.347 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.350 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.350 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.350 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.351 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.351 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.351 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.352 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.352 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.353 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.353 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.354 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.355 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:18.355 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.355 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.356 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.356 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.356 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.357 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.357 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.357 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.358 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.358 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.359 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.360 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:18.361 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.361 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.361 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.362 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.362 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.362 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.362 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.363 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.364 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.364 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.364 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.365 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:18.366 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.366 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.366 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.366 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.366 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.367 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.367 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.367 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.368 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.368 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.368 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.369 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:18.369 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.369 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.369 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.370 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.370 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.370 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.370 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.371 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.371 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.371 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.371 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.372 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.372 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.373 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:18.373 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.373 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.373 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.373 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.373 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.374 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.374 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.374 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.375 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.375 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.375 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.376 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:18.376 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.376 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:18.376 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.377 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.377 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:18.377 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jmNxEOgz4yiLDvZpPoUGl3lbqknNMpL", 2020-07-17T12:15:18.377 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.377 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 2 2020-07-17T12:15:18.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.378 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.378 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.14.11", 2020-07-17T12:15:18.378 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:10.801947Z" 2020-07-17T12:15:18.378 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.379 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:18.379 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.379 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.380 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.380 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.380 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.380 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.380 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.382 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.382 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.382 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.383 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.383 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:18.384 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.384 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.384 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.384 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.384 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.385 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.385 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.385 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.385 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.386 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.386 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.387 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:18.387 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-22 2020-07-17T12:15:18.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:18.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:18.389 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:18.431 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:18.431 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:18.673 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:18.673 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:18.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.674 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:18.674 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.675 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:18.675 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.675 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:18.676 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.676 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:18.677 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.677 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:18.678 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.678 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:18.678 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.679 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:18.679 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.680 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:18.680 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.680 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:18.680 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.14.11" 2020-07-17T12:15:18.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.681 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:18.681 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:18.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:18.683 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:18.683 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:18.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:18.683 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:18.683 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:18.685 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-22 2020-07-17T12:15:18.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:18.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:18.686 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:18.732 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:18.732 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:18.934 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:18.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.935 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:18.935 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.935 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.935 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.935 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.936 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.936 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.936 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.936 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.937 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.937 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.937 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.938 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:18.938 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.938 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.939 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.939 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.939 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.940 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.940 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.940 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.940 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.941 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.941 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.941 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.941 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.942 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:18.942 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.943 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.943 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.943 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.943 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.943 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.943 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.944 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.944 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.944 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.945 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.945 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.945 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.948 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:18.948 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.948 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.948 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.948 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.949 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.949 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.949 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.949 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.950 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.950 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.950 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.951 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:18.951 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.951 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.952 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.952 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.952 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.952 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.952 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.953 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.953 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.953 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.953 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.954 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.954 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.954 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:18.955 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.955 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.955 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.955 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.955 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.956 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.956 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.956 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.956 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.957 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.957 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.958 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:18.958 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.958 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.958 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.959 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.959 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.959 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.959 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.959 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.960 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.960 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.960 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.960 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.961 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:18.962 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.962 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.963 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.963 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.963 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.963 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.964 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.964 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.965 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.965 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.966 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.968 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:18.968 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.968 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:18.969 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.969 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.969 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:18.969 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jmNxEOgz4yiLDvZpPoUGl3lbqknNMpL", 2020-07-17T12:15:18.969 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.970 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 2 2020-07-17T12:15:18.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.970 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.970 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.14.11", 2020-07-17T12:15:18.971 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:10.805961Z" 2020-07-17T12:15:18.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.971 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:18.972 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.972 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.972 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.972 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.972 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.973 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.973 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.974 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.975 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.975 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.975 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.975 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.975 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:18.976 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:18.976 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:18.976 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:18.976 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:18.977 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:18.977 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:18.978 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:18.978 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:18.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.978 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:18.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:18.979 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:18.979 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:18.979 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:18.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:18.980 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:18.981 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-22 2020-07-17T12:15:18.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:18.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:18.982 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:19.059 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:19.060 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:19.267 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:19.268 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:19.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.268 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:19.268 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.269 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:19.269 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:19.273 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:19.275 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.11.5" 2020-07-17T12:15:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.277 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:19.277 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:19.278 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-22 2020-07-17T12:15:19.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:19.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:19.280 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:19.389 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:19.390 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:19.598 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "jmNxEOgz4yiLDvZpPoUGl3lbqknNMpL", 2020-07-17T12:15:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 2 2020-07-17T12:15:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.11.5", 2020-07-17T12:15:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:11.029892Z" 2020-07-17T12:15:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.635 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.635 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.635 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.636 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.636 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.637 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.637 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.638 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:19.638 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:19.638 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:19.638 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:19.639 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:19.639 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:19.639 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:19.639 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:19.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.639 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:19.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.640 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:19.640 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:19.640 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:19.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.641 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:19.641 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-22 2020-07-17T12:15:19.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:19.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:19.643 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:19.717 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:19.717 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:19.928 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:19.928 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:19.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.929 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:19.929 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.929 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:19.930 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.930 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:19.930 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.931 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:19.931 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.932 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:19.932 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.932 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:19.933 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.933 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:19.933 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.946 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:19.946 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.947 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:19.947 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.11.5" 2020-07-17T12:15:19.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.948 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:19.948 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:19.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:19.948 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:19.948 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:19.949 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:19.949 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:19.949 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:19.950 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-22 2020-07-17T12:15:19.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:19.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:19.952 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:20.002 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:20.002 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:20.221 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:20.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.222 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:20.223 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.223 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.225 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.225 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.225 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.226 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.226 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.226 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.226 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.227 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.227 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.228 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:20.228 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.228 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.228 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.229 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.229 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.229 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.229 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.229 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.230 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.230 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.230 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.231 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.231 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:20.231 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.231 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.232 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.232 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.232 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.232 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.232 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.233 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.233 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.233 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.233 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.234 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:20.235 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.235 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.235 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.235 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.235 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.235 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.236 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.237 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.237 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.237 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.238 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.239 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:20.239 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.239 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.239 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.239 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.240 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.240 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.240 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.240 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.241 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.241 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.241 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.242 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:20.242 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.242 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.243 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.243 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.243 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.243 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.243 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.244 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.244 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.244 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.244 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.245 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.245 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.257 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:20.257 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.257 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "jmNxEOgz4yiLDvZpPoUGl3lbqknNMpL", 2020-07-17T12:15:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 2 2020-07-17T12:15:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.11.5", 2020-07-17T12:15:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:11.031792Z" 2020-07-17T12:15:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:20.284 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:20.284 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-22 2020-07-17T12:15:20.285 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-22 zones={test-zone2, test-zone1} 2020-07-17T12:15:20.302 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:20.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.356 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:20.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:20.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:20.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:20.566 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-22 zones={test-zone2, test-zone1} 2020-07-17T12:15:20.567 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-22 zones={test-zone3, test-zone1} 2020-07-17T12:15:20.578 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:20.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.629 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:20.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:20.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:20.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:20.811 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-22 zones={test-zone3, test-zone1} 2020-07-17T12:15:20.813 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-22 zones={test-zone3, test-zone2} 2020-07-17T12:15:20.829 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:20.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.880 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:20.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:20.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:20.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:20.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:21.054 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-22 zones={test-zone3, test-zone2} 2020-07-17T12:15:21.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:21.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:21.056 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:21.098 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:21.099 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:21.306 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:21.307 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:21.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.307 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:21.308 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.309 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:21.310 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.311 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:21.311 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.312 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:21.312 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.313 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:21.313 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.314 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:21.315 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.316 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:21.316 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.317 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:21.317 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.318 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:21.319 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.8.5" 2020-07-17T12:15:21.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.320 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:21.320 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.321 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:21.322 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:21.322 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:21.323 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:21.324 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-23 2020-07-17T12:15:21.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:21.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:21.326 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:21.371 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:21.371 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:21.595 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "VSEBGGkMkwfd5mLnwt7UCJfsV2PrJpJ", 2020-07-17T12:15:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.8.5", 2020-07-17T12:15:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:11.115822Z" 2020-07-17T12:15:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:21.657 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:21.658 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-23 2020-07-17T12:15:21.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:21.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:21.659 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:21.701 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:21.930 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.938 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:21.938 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.8.5" 2020-07-17T12:15:21.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.939 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:21.939 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:21.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:21.940 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:21.940 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:21.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:21.940 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:21.941 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:21.942 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-23 2020-07-17T12:15:21.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:21.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:21.944 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:22.030 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:22.030 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:22.233 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.260 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.261 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.262 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.262 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.264 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:22.265 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.265 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.265 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.266 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.266 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.266 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.266 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.267 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.268 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.268 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.268 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.268 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.270 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:22.270 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.271 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.271 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.271 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.272 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.272 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.272 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.273 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.273 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.274 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.274 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.276 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:22.276 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.277 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.277 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.277 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.277 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.277 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.278 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.278 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.278 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.279 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.279 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.280 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:22.280 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.280 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.280 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.281 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.281 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.281 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.282 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.282 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.283 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.283 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.283 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.285 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:22.285 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.286 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:22.286 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.286 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.286 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:22.287 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "VSEBGGkMkwfd5mLnwt7UCJfsV2PrJpJ", 2020-07-17T12:15:22.287 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.288 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:22.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.288 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.289 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.8.5", 2020-07-17T12:15:22.289 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:11.117946Z" 2020-07-17T12:15:22.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.291 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:22.291 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.291 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.292 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.292 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.292 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.293 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.293 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.295 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.296 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.296 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.296 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.297 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:22.297 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:22.297 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:22.298 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:22.298 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:22.298 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:22.298 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:22.298 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:22.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.299 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:22.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.299 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:22.299 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:22.299 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.300 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:22.301 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-23 2020-07-17T12:15:22.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:22.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:22.302 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:22.377 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:22.588 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:22.588 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:22.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.588 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:22.589 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.589 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:22.589 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.590 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:22.590 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.591 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:22.591 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.592 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:22.592 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.592 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:22.593 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.594 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:22.594 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.595 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:22.595 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.596 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:22.596 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000002.8.5" 2020-07-17T12:15:22.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.597 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:22.598 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:22.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:22.598 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:22.598 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:22.598 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:22.599 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:22.599 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:22.600 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-23 2020-07-17T12:15:22.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:22.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:22.602 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:22.647 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:22.647 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:22.864 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:22.865 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.865 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:22.865 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.866 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.866 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.866 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.866 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.866 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.867 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.867 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.867 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.867 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.868 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.868 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.868 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.869 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:22.869 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.869 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.869 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.869 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.870 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.870 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.870 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.870 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.879 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:22.879 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.879 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.879 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.879 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.881 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.881 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.881 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.881 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.881 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.882 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.882 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:22.882 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.882 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.882 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.883 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.883 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.883 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.883 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.884 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.884 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.884 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.884 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.884 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.885 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.885 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.885 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:22.885 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.886 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.886 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.886 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.886 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.886 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.887 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.887 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.887 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.888 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.888 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.889 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:22.889 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.889 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.889 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.889 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.890 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.890 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.890 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.890 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.891 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.891 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.891 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.892 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:22.892 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.892 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:22.892 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.893 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.893 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.893 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.893 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000002.8.5", 2020-07-17T12:15:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:10.643044Z" 2020-07-17T12:15:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.895 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.895 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:22.895 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.897 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.918 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.918 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.918 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.919 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:22.919 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:22.919 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:22.920 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:22.920 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:22.920 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:22.920 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:22.920 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:22.920 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:22.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.921 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:22.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:22.921 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:22.921 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:22.922 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:22.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:22.923 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:22.941 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-23 2020-07-17T12:15:22.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:22.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:22.944 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:23.027 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:23.028 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:23.269 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:23.269 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:23.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.269 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:23.270 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.270 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.270 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:23.270 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.271 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.271 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:23.271 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.271 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.272 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:23.272 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.272 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:23.273 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.273 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.273 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:23.273 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.274 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:23.274 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.275 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:23.275 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.275 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:23.276 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000002.13.5" 2020-07-17T12:15:23.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.276 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:23.276 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.277 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.277 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:23.277 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.277 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:23.278 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:23.278 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:23.279 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-23 2020-07-17T12:15:23.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:23.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:23.281 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:23.323 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:23.323 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:23.535 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:23.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.536 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:23.536 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.536 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.537 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.537 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.537 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.538 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.538 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.539 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.539 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.540 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.540 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.540 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.540 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.541 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:23.550 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.550 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.551 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.551 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.551 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.553 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.553 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.553 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.554 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.554 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.555 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:23.555 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.555 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.556 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.556 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.556 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.556 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.557 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.557 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.559 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.559 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.560 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.561 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:23.562 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.562 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.562 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.563 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.563 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.563 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.563 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.564 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.565 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.565 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.565 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.566 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.566 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.567 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:23.567 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.567 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.568 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.569 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.569 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.570 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.570 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.571 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.571 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.572 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.572 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.574 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:23.574 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.574 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.575 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.575 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.575 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.576 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.576 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.577 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.577 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.578 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.578 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.582 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:23.582 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.583 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.583 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.583 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.584 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.584 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.584 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.585 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.585 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.586 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.586 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.586 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.587 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.588 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:23.588 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.589 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.589 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.589 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.590 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.590 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.590 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.591 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.592 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.592 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.593 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.593 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.595 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:23.595 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.595 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:23.598 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.598 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.599 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:23.599 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "VSEBGGkMkwfd5mLnwt7UCJfsV2PrJpJ", 2020-07-17T12:15:23.599 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.600 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:23.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.600 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.601 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.13.5", 2020-07-17T12:15:23.601 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:11.009031Z" 2020-07-17T12:15:23.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.603 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:23.603 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.603 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.604 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.604 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.604 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.604 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.605 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.607 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.607 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.607 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.608 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:23.609 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:23.609 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:23.609 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:23.609 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:23.609 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:23.609 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:23.610 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:23.610 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:23.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.611 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:23.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:23.611 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:23.611 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:23.612 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:23.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:23.613 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:23.614 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-23 2020-07-17T12:15:23.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:23.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:23.626 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:23.667 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:23.668 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:23.905 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:23.905 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:23.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.905 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:23.906 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.906 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:23.906 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.907 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:23.907 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.908 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:23.908 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.909 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:23.909 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.909 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:23.909 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.910 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:23.911 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.912 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:23.912 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.913 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:23.913 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000002.13.5" 2020-07-17T12:15:23.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.913 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:23.913 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:23.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:23.914 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:23.914 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:23.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:23.915 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:23.915 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:23.917 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-23 2020-07-17T12:15:23.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:23.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:23.919 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:23.963 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:23.963 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:24.160 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:24.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.160 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:24.160 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.161 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.161 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.161 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.161 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.161 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.162 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.162 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.162 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.162 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.163 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.164 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:24.164 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.164 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.164 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.164 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.165 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.165 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.165 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.165 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.166 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.166 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.166 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.167 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:24.168 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.168 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.168 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.169 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.169 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.169 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.170 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.170 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.171 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.171 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.171 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.173 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:24.173 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.176 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.177 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.177 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.177 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.178 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.178 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.179 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.179 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.180 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.180 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.195 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:24.196 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.196 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.196 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.196 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.196 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.197 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.197 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.198 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.198 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.198 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.199 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.199 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.199 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.199 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:24.200 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.200 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.200 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.200 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.200 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.201 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.201 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.201 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.202 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.202 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.202 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.203 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:24.203 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.203 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.203 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.204 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.204 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.204 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.205 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.206 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.206 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.207 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.207 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.218 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:24.218 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.218 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.218 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.218 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.219 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.219 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.219 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.219 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.220 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.220 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.220 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.220 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.220 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.221 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:24.221 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.221 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:24.222 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.222 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.222 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:24.222 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "VSEBGGkMkwfd5mLnwt7UCJfsV2PrJpJ", 2020-07-17T12:15:24.222 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.223 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:24.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.223 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.223 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000002.13.5", 2020-07-17T12:15:24.223 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:11.049996Z" 2020-07-17T12:15:24.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.224 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:24.224 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.225 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.226 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.226 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.226 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.226 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.227 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.228 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.228 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.229 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.229 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.230 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:24.230 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.230 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.230 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.230 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.231 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.231 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.231 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.231 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.232 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.232 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.232 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.233 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.233 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:24.233 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-23 2020-07-17T12:15:24.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:24.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:24.244 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:24.285 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:24.285 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:24.485 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:24.486 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:24.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.486 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:24.487 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.488 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:24.488 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.489 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:24.489 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.490 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:24.490 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.491 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:24.492 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.505 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:24.506 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.507 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:24.507 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.508 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:24.508 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.509 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:24.509 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.15.11" 2020-07-17T12:15:24.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.511 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:24.512 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.513 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:24.513 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:24.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.514 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:24.514 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:24.516 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-23 2020-07-17T12:15:24.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:24.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:24.518 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:24.608 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:24.609 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:24.803 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:24.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.804 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:24.804 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.804 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.804 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.805 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.805 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.805 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.805 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.806 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.806 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.806 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.806 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.807 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:24.808 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.808 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.809 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.809 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.809 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.809 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.809 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.810 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.810 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.810 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.811 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.812 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:24.812 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.812 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.812 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.812 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.812 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.813 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.813 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.813 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.814 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.814 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.814 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.815 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:24.815 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.815 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.816 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.816 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.816 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.816 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.816 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.817 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.817 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.817 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.818 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.819 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:24.819 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.819 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.819 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.819 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.820 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.820 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.820 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.820 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.821 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.821 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.821 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.822 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:24.822 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.823 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.823 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.823 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.823 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.823 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.824 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.824 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.824 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.824 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.825 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.826 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:24.826 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.826 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.826 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.826 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.834 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.835 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.835 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.835 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.836 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.836 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.836 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.837 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:24.837 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.859 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.859 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.860 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.860 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.860 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.860 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.861 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:24.863 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "VSEBGGkMkwfd5mLnwt7UCJfsV2PrJpJ", 2020-07-17T12:15:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.865 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.15.11", 2020-07-17T12:15:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:10.825929Z" 2020-07-17T12:15:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.867 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.868 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:24.875 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:24.876 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-23 2020-07-17T12:15:24.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:24.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:24.878 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:25.161 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:25.162 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:25.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.162 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:25.163 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.164 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:25.164 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.165 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:25.166 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.167 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:25.167 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.168 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:25.168 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.170 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:25.170 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.171 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:25.171 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.173 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:25.173 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.174 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:25.174 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.15.11" 2020-07-17T12:15:25.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.175 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:25.176 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:25.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:25.177 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:25.177 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:25.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:25.178 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:25.178 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:25.180 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-23 2020-07-17T12:15:25.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:25.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:25.182 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:25.230 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:25.230 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:25.455 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:25.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.455 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:25.455 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.456 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.456 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.456 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.456 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.457 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.457 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.459 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.460 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.460 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.460 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.461 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:25.461 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.462 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.462 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.462 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.462 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.462 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.463 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.463 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.463 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.464 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.464 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.465 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:25.465 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.465 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.465 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.465 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.474 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.474 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.475 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.475 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.476 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.476 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.477 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.477 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.477 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.478 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:25.478 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.479 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.479 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.479 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.480 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.480 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.480 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.481 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.481 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.481 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.483 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.484 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.484 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.484 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.484 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.486 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.488 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.488 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.488 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.489 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.489 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.489 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:25.490 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.490 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.490 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.490 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.490 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.490 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.491 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.491 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.491 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.492 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.492 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.493 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:25.493 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.493 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.493 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.494 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.494 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.494 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.494 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.495 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.495 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.495 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.495 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.496 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.496 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.496 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:25.496 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.497 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:25.497 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.497 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.497 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:25.497 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "VSEBGGkMkwfd5mLnwt7UCJfsV2PrJpJ", 2020-07-17T12:15:25.497 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.498 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:25.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.498 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.498 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.15.11", 2020-07-17T12:15:25.499 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:10.827828Z" 2020-07-17T12:15:25.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.500 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.501 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:25.501 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.502 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.502 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.502 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.502 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.502 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.503 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.504 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.504 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.504 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.505 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:25.506 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:25.506 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:25.506 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:25.506 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:25.507 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:25.507 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:25.507 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:25.507 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:25.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.508 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:25.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:25.508 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:25.508 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:25.508 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:25.508 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:25.509 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:25.509 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-23 2020-07-17T12:15:25.510 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-23 zones={test-zone2, test-zone1} 2020-07-17T12:15:25.522 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:25.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:25.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:25.576 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:25.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:25.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:25.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:25.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:25.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:25.742 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-23 zones={test-zone2, test-zone1} 2020-07-17T12:15:25.742 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-23 zones={test-zone3, test-zone1} 2020-07-17T12:15:25.752 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:25.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:25.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:25.799 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:25.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:25.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:25.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:25.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:25.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:25.962 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-23 zones={test-zone3, test-zone1} 2020-07-17T12:15:25.964 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-23 zones={test-zone3, test-zone2} 2020-07-17T12:15:25.976 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:26.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:26.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:26.028 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:26.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:26.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:26.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:26.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:26.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:26.200 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-23 zones={test-zone3, test-zone2} 2020-07-17T12:15:26.202 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-07-17T12:15:26.203 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-24 2020-07-17T12:15:26.235 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-25 2020-07-17T12:15:26.292 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-26 2020-07-17T12:15:26.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:26.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:26.448 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:26.497 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:26.497 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:26.697 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:15:26.722 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:26.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:26.723 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988126.433699_185.1", 2020-07-17T12:15:26.723 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:15:26.433699Z" 2020-07-17T12:15:26.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:26.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:26.724 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:26.725 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:15:26.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:26.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:26.726 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:26.726 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:15:26.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:26.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:26.727 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:26.727 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:15:26.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:26.728 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:26.731 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:15:26.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:26.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:26.733 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:26.775 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:26.775 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:27.013 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:27.013 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:15:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:15:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:15:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:15:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:27.014 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:27.015 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988126.351197_175.1", 2020-07-17T12:15:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:15:27.016 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.351197Z", 2020-07-17T12:15:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:27.017 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:27.018 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:27.019 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:27.020 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:27.021 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:27.022 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:27.023 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:27.024 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:27.032 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:15:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:15:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:15:27.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:27.033 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:27.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594988126.433699_185.1 target=1_1594988126.351197_175.1 2020-07-17T12:15:32.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:32.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:32.038 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:32.086 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:32.086 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:32.312 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:15:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:15:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:15:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:15:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.315 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988126.433699_185.1", 2020-07-17T12:15:32.317 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:15:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:32.318 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.433699Z", 2020-07-17T12:15:32.319 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:32.321 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:32.322 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:32.325 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:32.327 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:32.329 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:32.330 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:32.330 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:32.330 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:32.330 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:32.331 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:32.331 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.331 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:32.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.332 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:15:32.332 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:15:32.332 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:15:32.332 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.333 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:32.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:15:32.334 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:15:32.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:32.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:32.336 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:32.378 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:32.378 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:32.605 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:32.606 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:15:32.606 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:15:32.606 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:15:32.606 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:15:32.607 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:32.607 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:32.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:32.607 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:32.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:32.608 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:32.608 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:32.608 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:32.608 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988126.433699_185.1", 2020-07-17T12:15:32.608 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:32.609 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:15:32.609 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:32.609 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.433699Z", 2020-07-17T12:15:32.609 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:32.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:32.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:32.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:32.611 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:32.611 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:32.611 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:32.611 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:32.611 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:32.612 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:32.612 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:32.612 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:32.612 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:32.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:32.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:32.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:32.613 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:32.613 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:32.613 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:32.613 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:32.614 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:32.614 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:32.614 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:32.614 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:15:32.614 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:32.615 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:32.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:32.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:32.615 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:32.616 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:32.617 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:32.617 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:32.617 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:32.617 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:32.617 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:32.618 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:15:32.618 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:32.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:32.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:32.618 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:32.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:32.619 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:15:32.619 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:15:32.619 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:15:32.619 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:32.619 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:32.621 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:15:32.622 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:15:32.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:32.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:32.623 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:32.665 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:32.665 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:32.938 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:32.939 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:15:32.939 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:15:32.939 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:15:32.939 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:15:32.940 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:32.940 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.940 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:32.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:32.941 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:32.941 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:32.941 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:32.941 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988126.433699_185.1", 2020-07-17T12:15:32.941 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:32.941 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:15:32.942 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:32.942 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.433699Z", 2020-07-17T12:15:32.942 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.942 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:32.943 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:32.943 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:32.943 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:32.943 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:32.944 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:32.944 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:32.944 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:32.944 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:32.944 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:32.946 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:32.946 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:32.946 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:32.946 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:32.947 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:32.947 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:32.947 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:32.947 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:32.947 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.947 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:32.949 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:32.949 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:32.950 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:32.950 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:32.950 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:32.950 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:32.951 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:32.951 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:32.951 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:32.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.951 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:32.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:32.952 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:15:32.952 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:15:32.952 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:15:32.952 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:32.953 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:32.955 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:15:33.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:33.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:33.224 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:33.261 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:33.262 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:33.490 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:33.490 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:33.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.491 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:33.491 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.492 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:33.492 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.492 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:33.493 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.493 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:33.493 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.494 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:33.494 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.495 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:33.495 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.495 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:33.496 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.496 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:33.497 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.497 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:33.497 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.25.6" 2020-07-17T12:15:33.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.501 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:33.501 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:33.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:33.502 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:33.502 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:33.502 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:33.502 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:33.503 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:33.504 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-24 2020-07-17T12:15:33.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:33.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:33.506 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:33.552 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:33.552 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:33.760 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:33.760 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.761 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:33.761 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.761 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.762 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.762 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.762 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.763 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.763 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.764 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.765 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.765 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.766 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.768 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:33.768 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.769 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.769 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.770 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.770 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.770 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.771 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.771 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.772 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.772 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.773 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.774 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:33.775 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.775 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.776 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.776 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.776 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.776 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.777 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.778 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.779 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.779 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.780 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.781 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.781 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:33.781 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.782 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.782 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.782 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.783 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.783 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.783 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.784 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.785 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.785 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.785 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.787 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.787 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:33.788 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.789 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.789 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.789 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.790 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.790 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.790 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.791 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.792 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.792 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.792 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.792 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.793 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.794 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:33.794 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.794 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.795 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.795 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.795 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.796 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.796 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.797 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.797 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.798 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.798 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.799 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.799 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.800 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:33.801 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.801 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.801 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.802 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.802 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.802 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.803 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.803 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.804 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.804 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.805 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.805 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.805 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.807 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:33.807 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.808 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.808 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.808 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.809 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.809 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.810 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.810 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.811 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.812 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.812 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.812 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.813 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.814 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:33.814 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.814 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:33.815 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.815 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.815 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:33.816 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "null", 2020-07-17T12:15:33.816 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.817 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:33.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.817 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.818 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.25.6", 2020-07-17T12:15:33.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:33.127915Z" 2020-07-17T12:15:33.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.819 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.820 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.822 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.822 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.822 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.823 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.825 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.826 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.826 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.826 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.827 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.827 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:33.828 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:33.829 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:33.829 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:33.829 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:33.830 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:33.831 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:33.831 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:33.831 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:33.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.832 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:33.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:33.833 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:33.833 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:33.833 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:33.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.835 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:33.835 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:33.836 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-24 2020-07-17T12:15:33.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:33.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:33.839 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:33.880 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:33.881 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:34.124 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.124 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:34.125 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.127 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.130 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.25.6" 2020-07-17T12:15:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.132 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:34.132 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:34.134 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-24 2020-07-17T12:15:34.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:34.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:34.135 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:34.220 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:34.220 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:34.436 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:34.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.437 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:34.437 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.438 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.438 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.438 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.439 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.439 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.439 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.440 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.441 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.441 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.441 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.443 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:34.443 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.443 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.444 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.444 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.444 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.25.6", 2020-07-17T12:15:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.130068Z" 2020-07-17T12:15:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.501 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:34.501 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-24 2020-07-17T12:15:34.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:34.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:34.503 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:34.544 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:34.545 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:34.790 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:34.790 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:34.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.791 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:34.791 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.792 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:34.792 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.792 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:34.793 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.793 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:34.793 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.794 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:34.794 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.795 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:34.795 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.796 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:34.796 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.796 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:34.796 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.797 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:34.797 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.25.6" 2020-07-17T12:15:34.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.798 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:34.798 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:34.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:34.799 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:34.800 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:34.800 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:34.800 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:34.800 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:34.802 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-24 2020-07-17T12:15:34.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:34.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:34.804 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:34.849 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:34.850 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:35.061 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:35.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.061 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:35.062 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.062 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.062 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.062 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.062 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.063 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.063 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.063 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.064 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.064 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.064 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.065 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:35.065 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.065 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.066 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.066 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.066 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.066 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.066 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.067 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.067 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.067 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.068 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.068 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:35.069 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.069 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.069 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.069 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.069 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.072 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.073 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.073 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.074 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.074 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.074 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.076 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:35.076 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.076 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.076 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.076 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.077 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.077 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.077 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.077 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.078 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.078 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.078 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.079 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:35.080 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.080 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.080 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.080 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.080 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.080 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.081 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.082 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.082 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.082 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.083 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.084 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:35.084 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.084 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.084 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.084 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.085 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.085 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.085 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.085 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.086 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.086 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.086 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.087 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:35.087 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.088 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.088 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.088 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.088 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.089 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.089 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.089 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.090 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.090 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.090 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.091 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.091 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:35.092 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.092 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.093 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.106 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.106 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.107 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.107 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.107 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.108 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.108 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.108 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.109 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:35.109 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.109 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:35.109 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.110 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.110 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.110 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.110 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.111 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.111 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.111 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.25.6", 2020-07-17T12:15:35.111 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:32.960922Z" 2020-07-17T12:15:35.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:35.113 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.113 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.113 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.113 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.113 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.113 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.114 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.114 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.118 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.118 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.119 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.120 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:35.120 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:35.120 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:35.120 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:35.120 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:35.121 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:35.121 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:35.121 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:35.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.121 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:35.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.122 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:35.122 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:35.122 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.122 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.122 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.123 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:35.124 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-24 2020-07-17T12:15:35.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:35.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:35.126 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:35.168 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:35.170 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:35.405 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:35.406 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:35.406 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.406 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:35.407 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.407 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.407 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:35.407 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.408 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.408 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:35.408 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.409 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:35.409 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.410 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:35.410 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.411 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:35.411 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.411 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.411 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:35.412 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.414 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:35.414 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.415 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:35.415 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.29.12" 2020-07-17T12:15:35.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.416 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:35.416 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:35.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:35.417 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:35.417 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:35.417 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:35.417 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:35.417 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:35.418 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-24 2020-07-17T12:15:35.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:35.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:35.420 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:35.466 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:35.467 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:35.677 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:35.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.678 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:35.678 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.678 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.678 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.679 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.679 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.679 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.679 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.680 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.680 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.681 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.681 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.683 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:35.683 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.683 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.683 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.683 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.684 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.684 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.684 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.684 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.685 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.685 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.685 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.685 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.686 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:35.686 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.687 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.687 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.687 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.688 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.688 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.688 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.688 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.689 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.689 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.689 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.690 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:35.691 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.691 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.691 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.691 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.692 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.692 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.693 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.698 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.698 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.699 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.699 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.699 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.700 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:35.700 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.700 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.700 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.700 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.701 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.701 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.701 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.701 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.702 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.702 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.702 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.704 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:35.704 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.704 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.704 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.705 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.705 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.705 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.705 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.706 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.716 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.717 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.717 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.719 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:35.719 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.719 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.719 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.719 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.720 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.720 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.720 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.720 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.721 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.723 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:35.723 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.724 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.724 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.724 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.724 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.724 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.724 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.725 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.725 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.725 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.726 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.727 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:35.727 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.728 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:35.728 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.728 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.728 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:35.729 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:35.729 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.730 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:35.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.730 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.730 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.29.12", 2020-07-17T12:15:35.731 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.418033Z" 2020-07-17T12:15:35.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.732 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:35.732 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.732 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.733 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.733 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.733 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.733 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.733 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.735 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.735 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.735 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.735 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:35.736 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:35.737 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:35.737 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:35.737 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:35.738 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:35.738 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:35.738 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:35.739 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:35.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.739 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:35.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:35.740 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:35.740 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:35.740 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:35.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:35.741 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:35.742 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-24 2020-07-17T12:15:35.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:35.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:35.744 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:35.788 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:35.788 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:36.022 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:36.023 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:36.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.023 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:36.023 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.024 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:36.024 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.025 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:36.025 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.025 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:36.026 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.026 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:36.026 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.027 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:36.027 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.028 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:36.028 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.028 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:36.029 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.029 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:36.029 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.29.12" 2020-07-17T12:15:36.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.030 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:36.030 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:36.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:36.031 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:36.031 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:36.031 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:36.031 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:36.031 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:36.032 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-24 2020-07-17T12:15:36.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:36.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:36.036 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:36.080 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:36.308 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:36.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.308 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:36.308 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.309 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.309 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.309 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.309 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.309 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.310 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.310 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.310 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.310 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.311 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.312 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:36.312 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.312 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.312 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.312 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.312 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.313 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.313 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.313 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.314 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.314 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.314 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.315 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:36.315 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.315 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.315 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.316 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.316 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.316 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.316 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.316 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.317 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.317 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.317 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.328 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:36.329 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.329 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.329 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.329 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.329 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.331 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.331 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.331 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.332 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:36.332 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.332 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.332 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.333 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.333 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.333 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.333 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.333 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.334 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.334 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.334 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.335 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:36.335 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.335 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.336 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.336 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.336 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.336 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.336 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.338 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.338 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.338 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.338 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.339 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:36.340 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.340 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.340 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.340 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.340 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.341 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.341 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.341 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.341 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.342 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.342 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.343 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:36.343 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.343 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.343 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.344 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.344 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.344 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.344 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.344 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.345 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.345 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.346 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.347 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:36.347 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.347 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:36.348 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.348 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.348 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:36.348 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:36.348 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.349 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:36.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.349 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.349 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.29.12", 2020-07-17T12:15:36.350 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.421520Z" 2020-07-17T12:15:36.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.351 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:36.351 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.351 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.351 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.351 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.351 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.352 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.352 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.353 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.353 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.354 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.354 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.354 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.354 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.355 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:36.355 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:36.355 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:36.355 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:36.356 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:36.356 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:36.356 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:36.356 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:36.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.356 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:36.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.357 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:36.357 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:36.357 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:36.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.358 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:36.358 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-24 2020-07-17T12:15:36.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:36.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:36.360 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:36.435 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:36.436 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:36.684 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.27.12" 2020-07-17T12:15:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:36.693 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:36.694 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-24 2020-07-17T12:15:36.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:36.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:36.696 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:36.780 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:36.780 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:37.010 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:37.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.010 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:37.011 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.011 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.011 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.012 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.012 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.013 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.013 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.014 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.014 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.015 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.015 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.017 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:37.017 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.017 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.018 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.018 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.018 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.019 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.019 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.020 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.020 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.021 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.021 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.023 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:37.023 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.023 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.024 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.024 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.024 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.025 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.025 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.026 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.026 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.027 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.027 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.029 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:37.029 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.029 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.030 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.030 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.030 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.031 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.031 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.032 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.032 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.033 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.033 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.035 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:37.035 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.035 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.036 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.036 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.037 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.037 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.038 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.038 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.040 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.040 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.040 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.042 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:37.042 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.043 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.043 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.043 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.044 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.044 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.044 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.045 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.046 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.047 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.047 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.049 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:37.049 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.050 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.050 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.050 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.051 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.051 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.051 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.052 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.053 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.053 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.054 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.055 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:37.056 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.056 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.057 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.057 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.057 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.058 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.058 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.059 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.059 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.060 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.060 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.062 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:37.062 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.063 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:37.063 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.063 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.064 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:37.064 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:37.064 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.065 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:37.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.065 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.066 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.27.12", 2020-07-17T12:15:37.066 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.440637Z" 2020-07-17T12:15:37.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.068 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:37.068 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.068 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.069 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.069 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.069 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.070 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.070 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.072 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.073 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.073 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.073 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.075 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:37.075 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:37.076 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:37.076 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:37.076 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:37.076 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:37.076 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:37.076 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:37.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.077 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:37.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.077 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:37.077 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:37.078 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.078 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:37.079 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-24 2020-07-17T12:15:37.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:37.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:37.081 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:37.366 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:37.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.367 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.370 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:37.371 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.27.12" 2020-07-17T12:15:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:37.374 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:37.375 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:37.375 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:37.376 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-24 2020-07-17T12:15:37.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:37.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:37.378 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:37.424 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:37.424 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:37.650 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:37.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.651 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:37.651 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.651 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.651 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.651 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.651 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.652 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.652 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.652 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.653 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.653 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.653 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.655 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.655 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:37.655 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.655 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.655 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.656 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.656 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.656 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.656 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.657 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.657 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.657 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.657 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.660 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:37.663 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.664 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.664 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.664 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.664 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.664 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.665 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.665 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.665 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.666 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.666 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.666 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.666 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.667 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.667 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:37.667 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.667 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.667 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.667 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.668 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.668 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.668 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.668 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.669 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.669 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.669 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.669 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.669 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.670 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:37.670 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.670 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.671 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.671 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.671 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.671 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.671 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.672 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.672 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.672 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.672 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:37.673 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.674 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.675 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.677 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.678 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.679 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.679 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.679 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.679 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.680 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.680 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:37.680 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.680 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.681 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.682 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.683 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:37.684 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "null", 2020-07-17T12:15:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.685 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.27.12", 2020-07-17T12:15:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:33.399113Z" 2020-07-17T12:15:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.687 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.689 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:37.691 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:37.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:37.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:37.694 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:37.695 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-24 2020-07-17T12:15:37.695 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-24 zones={test-zone2, test-zone1} 2020-07-17T12:15:37.708 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:37.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:37.759 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:37.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:37.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:37.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-24 zones={test-zone2, test-zone1} 2020-07-17T12:15:37.837 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-24 zones={test-zone3, test-zone1} 2020-07-17T12:15:37.848 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:37.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:37.895 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:37.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:37.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:37.974 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-24 zones={test-zone3, test-zone1} 2020-07-17T12:15:37.975 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-24 zones={test-zone3, test-zone2} 2020-07-17T12:15:37.985 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:38.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:38.039 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:38.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:38.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:38.114 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-24 zones={test-zone3, test-zone2} 2020-07-17T12:15:38.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:38.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:38.116 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:38.390 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.397 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:38.398 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.15.12" 2020-07-17T12:15:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.402 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:38.402 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:38.402 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:38.403 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-25 2020-07-17T12:15:38.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:38.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:38.407 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:38.453 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:38.453 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:38.673 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:38.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.673 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:38.673 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.674 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.674 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.674 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.674 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.675 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.675 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.675 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.675 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.676 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.676 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.677 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:38.677 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.677 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.679 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.679 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.680 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.680 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.681 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.681 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.688 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.689 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.690 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.691 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:38.691 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.691 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.691 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.692 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.692 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.692 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.692 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.692 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.693 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.693 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.693 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.694 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:38.694 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.695 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.695 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.695 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.695 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.695 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.696 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.696 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.696 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.696 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.697 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.698 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:38.698 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.698 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.698 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.698 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.699 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.699 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.699 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.699 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.700 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.700 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.700 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.701 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:38.701 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.701 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.701 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.702 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.702 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.702 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.702 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.703 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.703 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.703 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.703 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.704 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.704 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.704 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:38.704 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.705 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.705 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.705 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.705 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.705 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.706 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.706 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.706 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.706 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.707 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.707 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.707 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.708 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:38.708 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.708 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.708 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.708 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.708 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.709 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.709 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.709 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.710 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.710 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.710 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.711 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:38.711 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.711 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:38.711 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.712 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.712 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:38.712 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "null", 2020-07-17T12:15:38.712 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.712 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:38.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.713 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.713 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.15.12", 2020-07-17T12:15:38.713 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:33.486191Z" 2020-07-17T12:15:38.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.714 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.714 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:38.714 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.714 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.715 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.715 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.715 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.715 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.715 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.717 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.717 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.717 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.718 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.719 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:38.719 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:38.719 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:38.719 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:38.719 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:38.719 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:38.720 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:38.720 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:38.720 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:38.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.720 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:38.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:38.721 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:38.721 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:38.722 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:38.722 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.722 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.722 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:38.722 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:38.723 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-25 2020-07-17T12:15:38.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:38.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:38.725 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:38.994 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.998 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:38.998 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.999 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:38.999 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:38.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:38.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:38.999 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:38.999 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.000 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:39.000 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.001 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:39.001 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.15.12" 2020-07-17T12:15:39.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.002 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:39.002 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.003 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:39.003 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.003 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:39.004 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:39.005 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-25 2020-07-17T12:15:39.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:39.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:39.008 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:39.088 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:39.088 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:39.311 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:39.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.311 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:39.312 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.312 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.312 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.313 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.313 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.313 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.314 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.315 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.316 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.316 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.316 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.318 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:39.318 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.319 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.319 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.319 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.320 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.320 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.320 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.322 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.322 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.322 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.323 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.323 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.323 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.324 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:39.324 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.324 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.325 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.325 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.325 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.325 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.325 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.326 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.326 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.326 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.327 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.327 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:39.328 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.328 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.328 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.328 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.328 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.329 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.329 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.329 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.330 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.330 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.330 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.334 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.334 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.334 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.335 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.335 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.336 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:39.336 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.337 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.337 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.337 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.337 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.337 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.338 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.338 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.338 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.339 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.340 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.341 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:39.341 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.341 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.341 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.341 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.342 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.342 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.342 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.342 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.343 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.343 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.343 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.344 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:39.344 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.344 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.345 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.345 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.346 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.346 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.346 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.347 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.347 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.347 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.347 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.350 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:39.350 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:39.352 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:39.352 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.353 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:39.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.353 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.354 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.15.12", 2020-07-17T12:15:39.354 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.485336Z" 2020-07-17T12:15:39.354 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.355 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:39.356 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.356 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.356 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.357 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.357 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.357 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.357 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.358 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.359 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.359 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.359 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.360 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:39.360 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:39.360 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:39.361 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:39.361 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:39.361 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:39.361 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:39.361 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:39.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.362 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:39.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.362 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:39.362 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:39.362 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.363 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.363 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.363 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.363 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:39.364 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-25 2020-07-17T12:15:39.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:39.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:39.365 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:39.449 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:39.449 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:39.677 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.15.12" 2020-07-17T12:15:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:39.688 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:39.689 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-25 2020-07-17T12:15:39.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:39.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:39.691 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:39.746 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:39.747 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:39.950 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:39.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:39.951 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:39.951 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:39.951 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:39.952 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:39.952 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:39.952 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:39.952 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:39.953 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:39.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.954 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:39.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.954 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:39.954 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:39.955 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.955 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.955 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:39.956 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:39.956 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:39.956 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:39.956 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:39.956 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:39.957 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:39.957 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:39.957 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:39.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.958 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:39.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.958 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:39.958 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:39.959 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.959 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.959 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:39.960 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:39.960 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:39.960 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:39.960 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:39.960 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:39.961 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:39.961 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:39.961 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:39.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.962 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:39.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.963 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:39.963 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:39.964 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.966 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:39.966 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:39.966 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:39.967 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:39.967 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:39.967 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:39.968 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:39.968 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:39.968 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:39.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.969 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:39.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.969 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:39.970 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:39.970 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.971 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.971 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:39.972 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:39.972 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:39.972 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:39.973 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:39.973 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:39.973 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:39.974 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:39.975 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:39.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.975 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:39.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.976 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:39.976 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:39.977 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:39.978 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:39.979 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:39.979 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:39.979 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:39.980 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:39.980 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:39.980 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:39.980 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:39.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.981 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:39.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.982 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:39.982 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:39.982 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:39.984 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:39.985 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:39.986 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:39.986 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:39.986 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:39.987 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:39.987 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:39.987 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:39.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.988 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:39.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.988 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:39.989 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:39.989 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:39.989 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.990 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:39.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:39.991 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:39.991 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:39.991 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:39.992 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:39.992 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:39.992 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:39.993 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:39.993 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:39.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:39.994 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:40.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.003 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:40.003 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:40.004 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.004 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.004 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.005 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:40.006 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:40.006 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:40.007 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:40.007 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:40.007 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:40.008 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:40.008 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:40.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.009 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:40.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.009 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:40.010 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.15.12", 2020-07-17T12:15:40.010 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:33.105672Z" 2020-07-17T12:15:40.010 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.010 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.012 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:40.012 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:40.013 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:40.013 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:40.013 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:40.014 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:40.014 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:40.014 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:40.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.015 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:40.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.016 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:40.016 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:40.016 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.017 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.017 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.017 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:40.018 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:40.018 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:40.018 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:40.018 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:40.018 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:40.018 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:40.019 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:40.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.019 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:40.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.019 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:40.020 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:40.020 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.020 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.020 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.020 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.021 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:40.021 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-25 2020-07-17T12:15:40.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:40.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:40.023 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:40.064 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:40.065 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:40.287 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:40.288 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:40.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.288 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:40.288 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.289 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:40.290 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.290 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.291 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:40.291 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.292 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.292 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:40.292 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.293 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:40.293 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.294 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:40.295 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.295 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.306 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:40.306 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.307 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:40.307 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:40.308 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.45.6" 2020-07-17T12:15:40.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.309 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:40.310 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.311 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:40.312 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:40.313 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-25 2020-07-17T12:15:40.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:40.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:40.316 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:40.363 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:40.363 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:40.565 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:40.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.566 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:40.566 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.566 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.566 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.567 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.567 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.567 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.567 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.567 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.568 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.568 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.568 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.568 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.569 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:40.569 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.570 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.570 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.570 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.570 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.570 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.570 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.571 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.571 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.571 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.572 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.572 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:40.573 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.573 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.573 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.573 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.573 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.582 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.582 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.582 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.583 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.583 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.583 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.584 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:40.584 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.585 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.585 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.585 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.585 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.585 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.587 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.588 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.589 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.590 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.590 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.592 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:40.592 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.593 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.593 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.593 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.593 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.594 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.594 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.595 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.595 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.596 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.596 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.596 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.598 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:40.598 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.598 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.599 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.599 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.599 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.600 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.600 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.601 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.601 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.602 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.602 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.603 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.604 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:40.604 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.604 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.605 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.605 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.605 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.618 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.619 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.619 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.620 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.620 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.620 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.621 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.621 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.622 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:40.622 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.622 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.623 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.623 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.623 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.623 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.623 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.624 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.624 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.624 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.625 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.625 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:40.626 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.626 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:40.626 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.626 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.626 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:40.627 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:40.627 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.627 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:40.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.628 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.628 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.45.6", 2020-07-17T12:15:40.628 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.213980Z" 2020-07-17T12:15:40.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.629 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:40.629 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.629 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.629 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.630 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.630 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.630 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.630 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.631 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.632 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.632 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.632 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:40.633 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:40.633 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:40.634 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:40.634 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:40.634 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:40.634 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:40.634 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:40.634 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:40.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.635 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:40.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:40.635 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:40.635 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:40.636 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:40.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:40.636 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:40.637 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-25 2020-07-17T12:15:40.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:40.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:40.639 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:40.681 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:40.682 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:40.910 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:40.910 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:40.911 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.911 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:40.911 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.912 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:40.912 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.913 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.913 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:40.914 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.915 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:40.915 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.916 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.923 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:40.924 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.925 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:40.926 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.927 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:40.928 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.929 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:40.929 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000003.45.6" 2020-07-17T12:15:40.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.930 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:40.931 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:40.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:40.932 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:40.934 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:40.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:40.940 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:40.941 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:40.942 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-25 2020-07-17T12:15:40.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:40.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:40.945 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:40.989 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:40.990 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:41.212 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:41.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.212 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:41.213 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.213 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.213 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.213 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.213 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.214 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.214 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.214 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.214 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.215 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.215 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.216 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:41.216 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.216 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.216 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.217 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.217 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.217 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.217 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.217 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.218 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.218 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.218 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.219 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.219 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:41.219 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.219 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.220 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.220 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.220 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.220 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.220 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.221 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.221 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.221 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.222 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.222 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.223 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:41.223 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.224 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.224 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.224 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.224 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.224 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.224 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.225 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.225 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.225 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.243 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.244 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:41.244 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.245 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.245 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.245 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.245 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.245 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.246 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.246 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.246 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.247 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.247 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.247 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.247 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.249 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:41.249 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.249 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.250 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.250 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.250 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.250 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.250 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.251 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.251 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.251 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.251 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.252 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.252 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.252 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:41.252 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.253 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.253 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.253 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.253 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.253 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.254 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.254 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.254 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.254 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.255 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.255 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.255 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.256 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:41.256 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.256 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.256 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.256 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.256 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.257 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.257 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.257 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.258 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.258 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.258 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.258 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.258 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.259 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:41.259 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.259 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:41.259 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.260 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.260 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:41.260 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:41.260 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.261 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:41.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.261 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.261 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.45.6", 2020-07-17T12:15:41.261 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.214286Z" 2020-07-17T12:15:41.261 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.262 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.262 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:41.262 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.263 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.263 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.263 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.263 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.263 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.263 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.265 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.265 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.266 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.266 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.267 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:41.267 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.267 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.268 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.268 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.268 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.268 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.268 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.269 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.269 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.269 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.270 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.270 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:41.271 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-25 2020-07-17T12:15:41.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:41.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:41.272 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:41.347 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:41.348 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:41.587 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:41.588 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:41.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.588 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:41.588 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.589 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:41.589 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.590 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:41.590 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.590 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:41.591 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.591 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:41.591 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.592 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:41.592 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.593 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:41.593 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.594 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:41.594 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.595 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:41.595 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.19.12" 2020-07-17T12:15:41.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.596 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:41.597 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.598 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:41.598 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:41.598 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.599 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:41.599 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:41.600 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-25 2020-07-17T12:15:41.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:41.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:41.603 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:41.684 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:41.685 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:41.895 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:41.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.897 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.898 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.898 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.898 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.898 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:41.900 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.901 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.902 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.903 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.905 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.906 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.906 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.906 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.906 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.907 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.908 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.908 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.909 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:41.909 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.909 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.909 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.909 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.909 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.910 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.910 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.910 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.911 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.911 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.911 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.913 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:41.913 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.913 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.913 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.913 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.914 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.914 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.915 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.915 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.916 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.916 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.916 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.918 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:41.918 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.918 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.918 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.918 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.918 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.919 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.919 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.919 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.921 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:41.921 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.921 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.922 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.922 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.922 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.922 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.922 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.923 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.923 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.924 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.924 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.924 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.924 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.925 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:41.925 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.925 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.925 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.926 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.926 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.926 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.926 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.927 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.927 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.927 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.927 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.928 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:41.929 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.929 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:41.929 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.929 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.929 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:41.930 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:41.930 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.930 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:41.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.931 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.931 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.19.12", 2020-07-17T12:15:41.931 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.482145Z" 2020-07-17T12:15:41.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.932 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:41.932 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.932 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.933 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.933 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.933 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.934 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.934 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.935 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.936 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.936 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.936 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:41.937 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:41.937 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:41.938 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:41.938 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:41.938 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:41.938 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:41.938 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:41.939 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:41.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.939 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:41.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:41.939 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:41.940 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:41.940 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:41.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:41.941 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:41.941 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-25 2020-07-17T12:15:41.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:41.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:41.945 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:42.025 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:42.026 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:42.255 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:42.255 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:42.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.255 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:42.256 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.256 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:42.256 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.257 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:42.257 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.258 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:42.258 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.258 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:42.259 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.259 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:42.259 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.260 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:42.260 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.261 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:42.261 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.261 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:42.264 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.19.12" 2020-07-17T12:15:42.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.265 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:42.265 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:42.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:42.266 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:42.266 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:42.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:42.266 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:42.266 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:42.267 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-25 2020-07-17T12:15:42.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:42.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:42.269 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:42.315 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:42.315 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:42.540 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:42.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.541 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:42.541 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.542 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.542 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.542 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.542 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.543 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.543 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.543 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.544 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.544 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.544 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.544 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.544 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.545 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:42.545 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.545 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.545 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.546 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.546 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.546 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.546 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.547 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.547 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.547 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.547 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.548 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.548 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:42.548 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.549 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.549 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.549 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.549 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.549 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.550 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.550 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.550 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.550 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.551 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.551 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.551 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.551 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.552 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:42.552 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.552 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.552 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.552 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.553 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.553 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.553 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.554 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.554 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.554 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.554 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.554 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.555 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:42.555 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.556 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.556 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.556 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.556 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.556 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.556 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.557 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.557 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.558 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.558 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.558 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.558 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.559 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:42.559 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.559 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.559 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.559 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.560 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.560 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.560 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.560 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.561 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.561 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.561 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.562 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:42.562 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.562 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.563 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.563 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.563 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.563 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.563 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.564 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.565 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.565 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.565 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.567 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:42.567 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.567 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.568 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.568 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.568 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.568 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.569 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.569 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.569 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.570 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.570 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.570 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.571 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.571 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:42.571 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.571 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:42.571 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.572 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.572 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:42.572 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "null", 2020-07-17T12:15:42.572 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.573 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:42.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.573 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.573 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.19.12", 2020-07-17T12:15:42.573 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:33.439880Z" 2020-07-17T12:15:42.573 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.574 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:42.574 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.575 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.575 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.575 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.575 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.575 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.576 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.577 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.577 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.577 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.578 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.578 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.578 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:42.579 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:42.580 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:42.580 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:42.580 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:42.580 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:42.581 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:42.581 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:42.581 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:42.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.582 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:42.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:42.583 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:42.583 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:42.583 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:42.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:42.584 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:42.585 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-25 2020-07-17T12:15:42.585 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-25 zones={test-zone2, test-zone1} 2020-07-17T12:15:42.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:42.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:42.647 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:42.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:42.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:42.724 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-25 zones={test-zone2, test-zone1} 2020-07-17T12:15:42.725 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-25 zones={test-zone3, test-zone1} 2020-07-17T12:15:42.733 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:42.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:42.783 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:42.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:42.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:42.850 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-25 zones={test-zone3, test-zone1} 2020-07-17T12:15:42.850 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-25 zones={test-zone3, test-zone2} 2020-07-17T12:15:42.863 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:42.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:42.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:42.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:42.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:42.979 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-25 zones={test-zone3, test-zone2} 2020-07-17T12:15:42.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:42.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:42.981 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:43.022 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:43.022 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:43.250 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:43.250 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:43.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.251 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:43.251 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.252 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:43.252 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.253 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:43.253 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.253 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:43.254 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.254 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:43.254 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.256 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:43.256 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.256 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:43.257 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.257 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:43.257 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.258 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:43.258 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.40.12" 2020-07-17T12:15:43.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.259 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:43.259 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.259 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:43.260 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:43.261 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:43.261 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:43.262 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-26 2020-07-17T12:15:43.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:43.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:43.264 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:43.310 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:43.311 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:43.529 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:43.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.531 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:43.531 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.531 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.531 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.531 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.532 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.532 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.532 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.532 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.533 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.533 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.533 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.549 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.550 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:43.550 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.551 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.551 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.551 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.551 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.552 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.552 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.552 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.552 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.553 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.553 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.553 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.553 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.565 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:43.565 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.565 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.566 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.566 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.566 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.566 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.566 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.567 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.567 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.567 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.567 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.568 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:43.569 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.569 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.569 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.569 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.569 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.580 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.580 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.581 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.582 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.582 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.582 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.584 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:43.584 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.585 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.585 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.585 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.586 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.586 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.586 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.587 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.587 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.588 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.588 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.589 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.600 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:43.600 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.600 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.600 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.601 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.601 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.601 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.601 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.602 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.602 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.602 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.602 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.603 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.603 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.603 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:43.603 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.604 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.604 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.604 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.604 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.604 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.604 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.605 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.605 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.605 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.606 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.607 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.607 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:43.607 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.608 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.608 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.608 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.608 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.608 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.609 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.609 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.609 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.609 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.610 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.611 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.611 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.611 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:43.611 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.612 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:43.612 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.612 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.612 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:43.612 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "null", 2020-07-17T12:15:43.612 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.613 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:43.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.613 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.614 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.40.12", 2020-07-17T12:15:43.614 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:33.527905Z" 2020-07-17T12:15:43.614 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.614 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.615 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:43.615 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.615 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.615 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.616 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.616 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.616 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.617 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.618 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.618 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:43.620 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:43.620 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:43.621 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:43.622 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:43.622 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:43.622 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:43.622 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:43.622 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:43.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.623 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:43.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:43.623 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:43.623 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:43.625 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:43.626 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=lbjidp-26 2020-07-17T12:15:43.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:43.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:43.628 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:43.673 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:43.674 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:43.910 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.40.12" 2020-07-17T12:15:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:43.920 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:43.921 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-26 2020-07-17T12:15:43.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:43.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:43.923 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:44.004 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:44.004 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:44.402 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:44.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.403 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:44.403 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.403 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.403 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.404 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.404 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.404 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.404 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.405 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.405 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.405 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.405 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.406 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:44.407 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.407 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.407 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.407 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.407 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.407 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.408 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.408 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.408 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.409 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.409 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.410 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:44.410 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.410 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.411 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.411 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.411 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.412 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.412 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.412 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.413 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.413 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.418 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.419 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:44.419 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.419 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.420 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.420 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.420 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.420 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.420 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.421 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.421 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.421 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.421 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.422 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:44.423 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.423 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.423 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.423 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.424 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.424 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.424 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.424 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.425 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.425 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.425 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.426 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:44.426 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.426 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.427 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.427 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.427 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.427 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.427 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.428 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.428 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.428 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.428 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.429 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:44.429 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.430 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.430 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.430 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.430 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.430 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.431 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.431 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.431 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.431 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.432 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.433 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:44.433 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.433 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.433 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.433 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.434 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.434 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.434 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.434 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.435 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.435 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.435 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.436 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:44.436 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.436 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:44.436 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.437 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.437 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:44.437 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:44.437 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.437 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:44.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.438 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.438 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.40.12", 2020-07-17T12:15:44.438 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.525974Z" 2020-07-17T12:15:44.438 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.439 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:44.439 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.439 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.440 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.440 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.440 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.440 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.440 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.442 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.442 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.442 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.442 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.443 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.443 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.443 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:44.444 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:44.444 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:44.444 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:44.444 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:44.445 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:44.445 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:44.445 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:44.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.445 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:44.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.446 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:44.446 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:44.446 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:44.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.447 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:44.447 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=lbjidp-26 2020-07-17T12:15:44.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:44.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:44.449 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:44.767 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:44.767 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:44.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.768 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:44.768 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.769 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:44.769 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.770 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:44.770 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.771 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:44.771 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.772 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:44.772 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.772 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:44.773 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.773 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:44.774 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.775 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:44.775 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.776 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:44.776 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000001.40.12" 2020-07-17T12:15:44.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.777 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:44.777 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:44.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:44.778 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:44.778 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:44.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:44.779 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:44.779 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:44.780 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-26 2020-07-17T12:15:44.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:44.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:44.782 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket lbjidp-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:44.826 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:44.827 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:45.036 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:45.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.036 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:45.036 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.037 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.037 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.037 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.037 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.037 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.038 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.038 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.039 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.039 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.040 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:45.040 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.040 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.040 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.040 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.043 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.043 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.043 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.047 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.047 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.047 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.048 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.048 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.048 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.048 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.049 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:45.049 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.049 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.049 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.049 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.051 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.051 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.051 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.053 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:45.053 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.053 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.053 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.053 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.054 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.054 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.054 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.056 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:45.056 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.057 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.057 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.057 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.057 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.057 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.057 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.058 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.058 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.058 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.059 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.061 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.061 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.061 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.063 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.063 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.063 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.064 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:45.064 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.065 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.065 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.065 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.065 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.065 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.066 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.066 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.066 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.067 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.067 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.068 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:45.068 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.069 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:45.069 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.069 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.069 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.069 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.070 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.070 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.071 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.071 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000001.40.12", 2020-07-17T12:15:45.071 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:33.194030Z" 2020-07-17T12:15:45.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.072 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:45.072 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.072 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.073 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.073 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.074 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.074 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.074 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.074 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.102 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.102 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.103 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.104 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:45.104 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:45.105 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:45.105 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:45.105 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:45.106 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:45.106 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:45.107 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:45.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.107 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:45.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.108 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:45.108 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:45.108 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.109 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.109 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.109 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.110 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:45.117 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=lbjidp-26 2020-07-17T12:15:45.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:45.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:45.119 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:45.197 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:45.252 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:45.473 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:45.474 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:45.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.475 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:45.475 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.476 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:45.476 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.477 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.477 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:45.478 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.479 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:45.479 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.480 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:45.480 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.481 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:45.481 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.484 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:45.484 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:45.485 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.30.12" 2020-07-17T12:15:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:45.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:45.489 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:45.489 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:45.489 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:45.490 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:45.490 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:45.491 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-26 2020-07-17T12:15:45.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:45.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:45.495 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:45.540 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:45.540 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:45.746 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:45.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.747 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:45.747 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.747 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.748 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.748 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.748 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.749 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.749 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.750 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.750 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.751 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.752 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.754 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:45.754 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.754 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.755 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.755 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.755 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.755 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.756 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.756 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.757 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.757 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.758 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.759 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:45.760 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.760 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.760 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.761 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.761 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.761 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.761 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.762 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.762 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.762 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.763 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.764 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:45.764 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.765 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.765 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.766 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.766 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.766 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.767 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.767 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.767 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.768 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.768 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.769 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:45.769 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.769 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.769 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.769 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.770 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.770 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.770 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.770 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.771 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.771 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.772 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.773 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:45.773 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.773 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.774 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.774 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.774 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.774 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.774 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.775 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.775 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.775 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.775 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.776 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:45.777 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.777 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.777 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.777 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.778 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.778 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.778 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.779 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.779 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.779 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.779 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.780 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.780 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.780 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:45.780 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.781 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.781 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.781 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.781 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.782 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.782 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.782 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.783 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.784 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.784 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.785 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:45.785 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.785 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:45.785 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.786 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.786 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:45.786 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:45.786 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.787 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:45.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.787 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.787 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.30.12", 2020-07-17T12:15:45.787 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.377537Z" 2020-07-17T12:15:45.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.788 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:45.789 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.789 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.789 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.789 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.789 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.789 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.790 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.791 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.791 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.791 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.792 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:45.794 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:45.794 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:45.794 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:45.794 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:45.795 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:45.796 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:45.796 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.797 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:45.799 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:45.799 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=lbjidp-26 2020-07-17T12:15:45.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:45.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:45.801 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:45.843 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:45.844 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:46.076 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:46.076 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:46.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.077 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:46.078 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.078 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.079 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:46.079 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.079 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:46.080 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.080 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:46.081 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.081 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:46.081 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.082 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:46.082 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.083 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:46.083 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.083 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:46.084 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.084 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:46.085 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "00000000001.30.12" 2020-07-17T12:15:46.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.086 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:46.086 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:46.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:46.086 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:46.086 INFO:teuthology.orchestra.run.smithi052.stdout: "val": "" 2020-07-17T12:15:46.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:46.087 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:46.087 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:46.088 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-26 2020-07-17T12:15:46.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:46.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:46.091 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket lbjidp-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:46.135 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:46.136 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:46.352 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:46.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.353 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:46.353 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.354 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.354 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.354 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.355 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.355 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.355 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.356 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.357 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.357 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.357 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.363 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:46.363 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.363 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.363 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.364 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.364 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.364 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.364 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.365 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.365 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.365 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.370 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.372 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:46.373 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.373 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.374 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.374 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.374 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.374 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.375 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.375 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.375 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.376 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.376 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.378 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:46.378 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.379 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.379 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.379 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.379 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.380 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.380 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.380 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.381 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.381 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.382 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.382 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.382 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.383 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:46.383 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.383 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.383 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.383 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.384 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.384 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.384 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.384 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.385 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.385 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.385 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.385 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.386 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:46.387 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.387 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.387 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.387 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.387 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.388 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.388 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.389 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.389 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.389 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.390 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.390 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.390 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.391 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:46.391 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.391 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.392 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.392 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.392 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.392 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.392 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.393 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.393 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.393 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.394 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.394 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:46.395 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.395 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.395 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.395 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.395 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.396 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.396 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.396 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.397 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.397 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.398 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.399 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.400 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:46.400 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.400 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:46.401 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.402 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.402 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:46.402 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:46.402 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.404 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:46.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.404 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.404 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000001.30.12", 2020-07-17T12:15:46.404 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.461988Z" 2020-07-17T12:15:46.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.405 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.406 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:46.406 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.406 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.406 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.407 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.407 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.408 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.408 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.410 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.411 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.411 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.411 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.412 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:46.412 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:46.412 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:46.412 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:46.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.414 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:46.414 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:46.414 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:46.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.415 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:46.415 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=lbjidp-26 2020-07-17T12:15:46.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:46.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:46.417 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:46.494 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:46.495 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:46.738 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.42.6" 2020-07-17T12:15:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:46.758 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:46.761 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-26 2020-07-17T12:15:46.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:46.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:46.763 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:46.845 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:46.846 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:47.059 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:47.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.060 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:47.060 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.060 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.060 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.063 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.063 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.063 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.063 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.065 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.065 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.065 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.065 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.073 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.073 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.073 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.074 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.075 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:47.075 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.075 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.075 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.076 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.076 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.076 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.076 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.077 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.077 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.077 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.077 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.079 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:47.079 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.079 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.079 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.079 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.080 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.080 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.080 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.081 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.081 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.081 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.082 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.083 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:47.083 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.083 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.083 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.084 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.084 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.084 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.084 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.085 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.085 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.085 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.085 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.086 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.087 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:47.087 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.087 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.088 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.089 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.089 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.089 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.089 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.090 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.094 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.094 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.094 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.095 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:47.095 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.096 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.096 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.096 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.096 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.096 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.096 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.097 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.097 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.097 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.098 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.099 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:47.099 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.099 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.099 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.099 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.099 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.100 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.100 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.100 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.101 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.101 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.101 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.102 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:47.102 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.102 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:15:47.102 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.103 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.103 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "obj", 2020-07-17T12:15:47.103 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "null", 2020-07-17T12:15:47.104 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.105 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 1 2020-07-17T12:15:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "00000000003.42.6", 2020-07-17T12:15:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:33.144804Z" 2020-07-17T12:15:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:47.109 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.110 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.112 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.114 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "init", 2020-07-17T12:15:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "full_marker": { 2020-07-17T12:15:47.117 INFO:teuthology.orchestra.run.smithi146.stdout: "position": { 2020-07-17T12:15:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "", 2020-07-17T12:15:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "instance": "", 2020-07-17T12:15:47.118 INFO:teuthology.orchestra.run.smithi146.stdout: "ns": "" 2020-07-17T12:15:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "count": 0 2020-07-17T12:15:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.119 INFO:teuthology.orchestra.run.smithi146.stdout: "inc_marker": { 2020-07-17T12:15:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "position": "", 2020-07-17T12:15:47.120 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.126 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:47.126 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=lbjidp-26 2020-07-17T12:15:47.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:47.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'lbjidp-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:47.128 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket lbjidp-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:47.197 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:47.198 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:47.423 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.424 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.426 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:47.429 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.430 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 4, 2020-07-17T12:15:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 5, 2020-07-17T12:15:47.432 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.438 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 6, 2020-07-17T12:15:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 7, 2020-07-17T12:15:47.439 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 8, 2020-07-17T12:15:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "00000000003.42.6" 2020-07-17T12:15:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 9, 2020-07-17T12:15:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:47.441 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 10, 2020-07-17T12:15:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: "val": "" 2020-07-17T12:15:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:47.442 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:47.442 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:47.443 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-26 2020-07-17T12:15:47.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:47.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'lbjidp-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:47.445 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket lbjidp-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:47.708 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:15:47.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.708 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:47.709 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.709 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.709 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.709 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.710 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.710 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.710 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.710 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.711 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.711 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.711 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.712 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.713 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.713 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.713 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.713 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.713 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.714 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.714 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.714 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.714 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.715 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.715 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.715 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:47.716 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.716 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.716 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.717 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.717 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.727 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.728 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.728 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.729 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.729 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.729 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.729 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.729 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.730 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:47.730 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.730 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.731 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.731 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.731 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.731 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.732 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.732 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.732 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.733 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.733 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 4, 2020-07-17T12:15:47.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.740 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.740 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.740 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.741 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.741 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.741 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.741 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.742 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.742 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.742 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.743 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.743 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 5, 2020-07-17T12:15:47.743 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.744 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.744 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.744 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.744 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.744 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.745 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.745 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.746 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.746 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.746 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.747 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 6, 2020-07-17T12:15:47.747 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.748 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.748 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.748 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.748 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.748 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.749 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.749 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.749 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.749 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.750 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.750 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.750 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.751 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 7, 2020-07-17T12:15:47.751 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.751 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.751 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.751 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.752 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.752 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.752 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.752 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.753 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.753 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.753 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.762 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 8, 2020-07-17T12:15:47.762 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.763 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:15:47.763 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.763 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.763 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "obj", 2020-07-17T12:15:47.763 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "null", 2020-07-17T12:15:47.764 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.764 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 1 2020-07-17T12:15:47.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.764 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.765 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "00000000003.42.6", 2020-07-17T12:15:47.765 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:33.144804Z" 2020-07-17T12:15:47.765 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.765 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.766 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 9, 2020-07-17T12:15:47.766 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.766 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.766 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.767 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.767 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.767 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.767 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.770 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.771 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.771 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.771 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.772 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:47.772 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 10, 2020-07-17T12:15:47.772 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:47.772 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "init", 2020-07-17T12:15:47.773 INFO:teuthology.orchestra.run.smithi052.stdout: "full_marker": { 2020-07-17T12:15:47.773 INFO:teuthology.orchestra.run.smithi052.stdout: "position": { 2020-07-17T12:15:47.773 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "", 2020-07-17T12:15:47.773 INFO:teuthology.orchestra.run.smithi052.stdout: "instance": "", 2020-07-17T12:15:47.773 INFO:teuthology.orchestra.run.smithi052.stdout: "ns": "" 2020-07-17T12:15:47.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.774 INFO:teuthology.orchestra.run.smithi052.stdout: "count": 0 2020-07-17T12:15:47.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:47.774 INFO:teuthology.orchestra.run.smithi052.stdout: "inc_marker": { 2020-07-17T12:15:47.774 INFO:teuthology.orchestra.run.smithi052.stdout: "position": "", 2020-07-17T12:15:47.775 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2020-07-17T12:15:47.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:47.775 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:15:47.776 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=lbjidp-26 2020-07-17T12:15:47.776 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-26 zones={test-zone2, test-zone1} 2020-07-17T12:15:47.790 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:47.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:47.839 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:47.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:47.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:47.908 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-26 zones={test-zone2, test-zone1} 2020-07-17T12:15:47.908 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-26 zones={test-zone3, test-zone1} 2020-07-17T12:15:47.916 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:47.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:47.967 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:47.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:48.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:48.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-26 zones={test-zone3, test-zone1} 2020-07-17T12:15:48.038 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-26 zones={test-zone3, test-zone2} 2020-07-17T12:15:48.048 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:15:48.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:48.096 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:15:48.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-17T12:15:48.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-17T12:15:48.173 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-26 zones={test-zone3, test-zone2} 2020-07-17T12:15:48.175 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-07-17T12:15:48.175 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-27 2020-07-17T12:15:48.214 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-28 2020-07-17T12:15:48.269 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-29 2020-07-17T12:15:48.534 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-07-17T12:15:48.535 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-30 2020-07-17T12:15:48.563 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-31 2020-07-17T12:15:48.623 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-32 2020-07-17T12:15:48.890 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-07-17T12:15:48.891 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-33 2020-07-17T12:15:48.914 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-34 2020-07-17T12:15:48.963 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-35 2020-07-17T12:15:51.198 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-07-17T12:15:51.199 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-36 2020-07-17T12:15:51.242 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-37 2020-07-17T12:15:51.311 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-38 2020-07-17T12:15:51.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:51.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:51.370 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:51.426 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:51.426 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:51.602 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:15:51.633 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988151.342294_251.1", 2020-07-17T12:15:51.634 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:15:51.342294Z" 2020-07-17T12:15:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:51.635 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:15:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:51.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:15:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:51.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:51.639 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:15:51.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:51.641 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:51.642 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:15:51.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:51.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:51.645 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:51.692 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:51.693 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:51.932 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:15:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:15:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:15:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:15:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988151.342294_251.1", 2020-07-17T12:15:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:15:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.342294Z", 2020-07-17T12:15:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:15:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:15:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:15:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:51.959 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:51.960 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:15:51.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:15:51.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:51.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:51.963 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:52.004 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:52.005 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:52.229 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:15:52.229 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:15:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:15:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:15:52.230 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:52.231 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:52.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:52.233 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988151.342294_251.1", 2020-07-17T12:15:52.234 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:15:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:52.235 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.342294Z", 2020-07-17T12:15:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:52.236 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:52.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:52.242 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:52.243 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:52.244 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:52.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:52.246 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:52.247 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:15:52.248 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:52.249 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:52.250 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:15:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:52.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:15:52.252 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:15:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:15:52.253 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:52.253 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:52.254 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:15:52.255 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:15:52.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:52.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:52.256 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:52.298 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:52.298 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:52.568 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:15:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:15:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:15:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:15:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988151.342294_251.1", 2020-07-17T12:15:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:15:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.342294Z", 2020-07-17T12:15:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:15:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:15:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:15:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:52.581 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:52.582 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:15:52.939 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-07-17T12:15:52.940 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-39 2020-07-17T12:15:52.974 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-40 2020-07-17T12:15:53.034 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-41 2020-07-17T12:15:53.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:53.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:53.102 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:53.145 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:53.145 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:53.300 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=34954458-202e-48c3-864b-c00b15f41dd5 2020-07-17T12:15:53.327 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:15:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988153.060681_263.1", 2020-07-17T12:15:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:15:53.060681Z" 2020-07-17T12:15:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:06:43.895710Z" 2020-07-17T12:15:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:15:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:15:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:53.332 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:15:53.333 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:15:53.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:53.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:53.370 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:53.409 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:53.410 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:53.638 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:53.639 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:15:53.639 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:15:53.639 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:15:53.639 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:15:53.640 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:53.640 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:53.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:53.640 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:53.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:53.641 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:53.641 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:53.641 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:53.641 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988153.060681_263.1", 2020-07-17T12:15:53.641 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:53.641 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:15:53.642 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:53.642 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.060681Z", 2020-07-17T12:15:53.642 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:53.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:53.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:53.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:53.643 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:53.643 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:53.643 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:53.644 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:53.644 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:53.644 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:53.644 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:53.644 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:53.645 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:53.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:53.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:53.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:53.645 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:53.645 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:53.646 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:53.646 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:53.646 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:53.646 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:53.646 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:53.647 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:53.647 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:53.647 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:53.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:53.647 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:53.648 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:53.648 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:53.648 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:53.648 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:53.648 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:53.649 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:53.649 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:53.649 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:53.649 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:53.649 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:53.650 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:53.650 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:53.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:53.658 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:15:53.658 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:15:53.658 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:15:53.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:53.659 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:53.660 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:15:53.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:15:53.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:53.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:53.662 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:53.932 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:15:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:15:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:15:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:15:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:15:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:15:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:15:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988153.060681_263.1", 2020-07-17T12:15:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:15:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.060681Z", 2020-07-17T12:15:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:15:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:15:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:15:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:15:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:15:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:15:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:15:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:15:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:15:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:15:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:15:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:15:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:15:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:15:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:15:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:15:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:15:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:15:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:15:53.947 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:15:53.949 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:15:53.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:15:53.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:53.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:15:53.952 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:15:54.000 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:15:54.000 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:15:54.272 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:15:54.272 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:15:54.273 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:15:54.273 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:15:54.273 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:15:54.274 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:15:54.274 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:54.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:54.275 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:15:54.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:54.275 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:15:54.276 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:54.280 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:54.281 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988153.060681_263.1", 2020-07-17T12:15:54.281 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:54.281 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:15:54.282 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:54.282 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.060681Z", 2020-07-17T12:15:54.282 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:54.283 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:54.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:54.283 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:54.284 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:15:54.284 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:54.284 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:54.285 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:15:54.285 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:54.285 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:54.286 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:54.287 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:15:54.287 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:54.287 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:15:54.289 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:54.289 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:54.289 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:54.290 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:54.290 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:54.290 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:54.291 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:54.291 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:54.291 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:54.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:54.292 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:15:54.292 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:15:54.293 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:15:54.293 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:15:54.293 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:15:54.293 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:15:54.294 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:15:54.294 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:15:54.294 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:15:54.294 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:15:54.294 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:54.294 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:54.295 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:15:54.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:15:54.295 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:15:54.295 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:15:54.295 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:15:54.296 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:15:54.296 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:15:54.305 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:15:54.306 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-07-17T12:15:54.307 INFO:teuthology.orchestra.run:waiting for 300 2020-07-17T12:15:54.308 INFO:tasks.rgw.c1.client.1.smithi146.stdout:2020-07-17T12:15:54.301+0000 7fcd509ec700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 16304) UID: 0 2020-07-17T12:15:54.308 INFO:tasks.rgw.c1.client.1.smithi146.stdout:2020-07-17T12:15:54.301+0000 7fcd680c0100 -1 shutting down 2020-07-17T12:16:00.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-07-17T12:16:00.408 INFO:tasks.rgw.c1.client.1:Stopped 2020-07-17T12:16:00.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:16:00.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:16:00.411 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:16:00.450 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:16:00.450 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:16:00.480 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:16:00.467+0000 7fe6bd23b0c0 0 NOTICE: overriding master zone: bc359d1d-af48-401e-ac43-335f84c3f1d9 2020-07-17T12:16:00.480 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:16:00.467+0000 7fe6bd23b0c0 0 failed reading obj info from .rgw.root:zone_info.0c5f4c70-7e48-49ca-abec-dc0f91059509: (2) No such file or directory 2020-07-17T12:16:00.480 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:16:00.467+0000 7fe6bd23b0c0 0 WARNING: could not read zone params for zone id=0c5f4c70-7e48-49ca-abec-dc0f91059509 name=test-zone3 2020-07-17T12:16:00.482 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:16:00.471+0000 7fe6bd23b0c0 0 failed reading obj info from .rgw.root:zone_info.bc359d1d-af48-401e-ac43-335f84c3f1d9: (2) No such file or directory 2020-07-17T12:16:00.482 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:16:00.471+0000 7fe6bd23b0c0 0 WARNING: could not read zone params for zone id=bc359d1d-af48-401e-ac43-335f84c3f1d9 name=test-zone1 2020-07-17T12:16:00.493 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:16:00.494 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:16:00.494 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2020-07-17T12:16:00.494 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-07-17T12:16:00.494 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone2.rgw.control", 2020-07-17T12:16:00.495 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-07-17T12:16:00.495 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-07-17T12:16:00.495 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone2.rgw.log", 2020-07-17T12:16:00.495 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-07-17T12:16:00.495 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-07-17T12:16:00.495 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-07-17T12:16:00.496 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-07-17T12:16:00.496 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-07-17T12:16:00.496 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-07-17T12:16:00.496 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-07-17T12:16:00.496 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-07-17T12:16:00.497 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-07-17T12:16:00.497 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2020-07-17T12:16:00.497 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:16:00.497 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:16:00.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.497 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2020-07-17T12:16:00.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.498 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2020-07-17T12:16:00.498 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:16:00.498 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-07-17T12:16:00.498 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2020-07-17T12:16:00.499 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2020-07-17T12:16:00.499 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-07-17T12:16:00.499 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2020-07-17T12:16:00.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:16:00.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.499 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-07-17T12:16:00.500 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2020-07-17T12:16:00.500 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:16:00.500 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:16:00.500 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.500 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca" 2020-07-17T12:16:00.501 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:16:00.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:16:00.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:16:00.504 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:16:00.586 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:16:00.587 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:16:00.864 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:16:00.864 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "159da1b7-e55f-4be6-b11f-00c3d4a2d701", 2020-07-17T12:16:00.865 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 1, 2020-07-17T12:16:00.865 INFO:teuthology.orchestra.run.smithi052.stdout: "predecessor_uuid": "34954458-202e-48c3-864b-c00b15f41dd5", 2020-07-17T12:16:00.865 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": [ 2020-07-17T12:16:00.865 INFO:teuthology.orchestra.run.smithi052.stdout: "1_1594988153.060681_263.1", 2020-07-17T12:16:00.865 INFO:teuthology.orchestra.run.smithi052.stdout: "1_1594987603.895710_86.1", 2020-07-17T12:16:00.866 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2020-07-17T12:16:00.866 INFO:teuthology.orchestra.run.smithi052.stdout: "" 2020-07-17T12:16:00.866 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.866 INFO:teuthology.orchestra.run.smithi052.stdout: "period_map": { 2020-07-17T12:16:00.867 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "159da1b7-e55f-4be6-b11f-00c3d4a2d701", 2020-07-17T12:16:00.867 INFO:teuthology.orchestra.run.smithi052.stdout: "zonegroups": [ 2020-07-17T12:16:00.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.867 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "test-zonegroup", 2020-07-17T12:16:00.867 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zonegroup", 2020-07-17T12:16:00.867 INFO:teuthology.orchestra.run.smithi052.stdout: "api_name": "", 2020-07-17T12:16:00.868 INFO:teuthology.orchestra.run.smithi052.stdout: "is_master": "true", 2020-07-17T12:16:00.868 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:16:00.868 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:16:00.868 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.868 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames": [], 2020-07-17T12:16:00.869 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames_s3website": [], 2020-07-17T12:16:00.869 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:16:00.869 INFO:teuthology.orchestra.run.smithi052.stdout: "zones": [ 2020-07-17T12:16:00.869 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.870 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:16:00.870 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone3", 2020-07-17T12:16:00.870 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:16:00.870 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8001" 2020-07-17T12:16:00.870 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.871 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:16:00.871 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:16:00.871 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:16:00.871 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:16:00.871 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:16:00.872 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:16:00.872 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:16:00.872 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:16:00.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.873 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:16:00.873 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone4", 2020-07-17T12:16:00.874 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:16:00.874 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8001" 2020-07-17T12:16:00.874 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.885 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:16:00.886 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:16:00.886 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:16:00.886 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:16:00.887 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "pubsub", 2020-07-17T12:16:00.887 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "false", 2020-07-17T12:16:00.887 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [ 2020-07-17T12:16:00.888 INFO:teuthology.orchestra.run.smithi052.stdout: "test-zone1" 2020-07-17T12:16:00.888 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.888 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:16:00.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.889 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:16:00.889 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone1", 2020-07-17T12:16:00.890 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:16:00.890 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:16:00.890 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.891 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:16:00.891 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:16:00.892 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:16:00.892 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:16:00.892 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:16:00.893 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:16:00.893 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:16:00.893 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:16:00.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.894 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:16:00.895 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2020-07-17T12:16:00.895 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:16:00.896 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2020-07-17T12:16:00.896 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.897 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:16:00.897 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:16:00.897 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:16:00.897 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:16:00.898 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:16:00.898 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:16:00.898 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:16:00.899 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:16:00.899 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:16:00.899 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.899 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_targets": [ 2020-07-17T12:16:00.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.900 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default-placement", 2020-07-17T12:16:00.900 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2020-07-17T12:16:00.901 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": [ 2020-07-17T12:16:00.901 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD" 2020-07-17T12:16:00.901 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:16:00.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:16:00.902 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.902 INFO:teuthology.orchestra.run.smithi052.stdout: "default_placement": "default-placement", 2020-07-17T12:16:00.902 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:16:00.903 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_policy": { 2020-07-17T12:16:00.903 INFO:teuthology.orchestra.run.smithi052.stdout: "groups": [] 2020-07-17T12:16:00.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:16:00.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:16:00.904 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:16:00.905 INFO:teuthology.orchestra.run.smithi052.stdout: "short_zone_ids": [ 2020-07-17T12:16:00.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.906 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:16:00.907 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 2627409927 2020-07-17T12:16:00.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.908 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:16:00.908 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3854808555 2020-07-17T12:16:00.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.908 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.909 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:16:00.909 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 340514820 2020-07-17T12:16:00.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:16:00.910 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:16:00.910 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3926357550 2020-07-17T12:16:00.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:16:00.911 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:16:00.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.911 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zonegroup": "test-zonegroup", 2020-07-17T12:16:00.912 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:16:00.912 INFO:teuthology.orchestra.run.smithi052.stdout: "period_config": { 2020-07-17T12:16:00.912 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_quota": { 2020-07-17T12:16:00.912 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2020-07-17T12:16:00.913 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2020-07-17T12:16:00.913 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2020-07-17T12:16:00.913 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2020-07-17T12:16:00.914 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2020-07-17T12:16:00.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.914 INFO:teuthology.orchestra.run.smithi052.stdout: "user_quota": { 2020-07-17T12:16:00.914 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2020-07-17T12:16:00.915 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2020-07-17T12:16:00.915 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2020-07-17T12:16:00.915 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2020-07-17T12:16:00.916 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2020-07-17T12:16:00.916 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:16:00.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:16:00.917 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:16:00.917 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_name": "test-realm", 2020-07-17T12:16:00.918 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 3 2020-07-17T12:16:00.918 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:16:00.918 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-07-17T12:16:05.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-07-17T12:17:00.920 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-42 2020-07-17T12:17:00.994 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-43 2020-07-17T12:17:01.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:17:01.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:17:01.035 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:17:01.074 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:17:01.077 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:17:01.264 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=159da1b7-e55f-4be6-b11f-00c3d4a2d701 2020-07-17T12:17:01.299 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:17:01.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:17:01.300 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988221.023402_317.1", 2020-07-17T12:17:01.300 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:17:01.023402Z" 2020-07-17T12:17:01.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:17:01.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:17:01.300 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:17:01.301 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "0.000000" 2020-07-17T12:17:01.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:17:01.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:17:01.301 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:17:01.301 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "0.000000" 2020-07-17T12:17:01.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:17:01.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:17:01.302 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:17:01.302 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "0.000000" 2020-07-17T12:17:01.302 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:17:01.302 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:17:01.304 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-07-17T12:17:01.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:17:01.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:17:01.306 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:17:01.348 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:17:01.350 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:17:01.579 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:17:01.579 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:17:01.580 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:17:01.580 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:17:01.580 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:17:01.580 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "159da1b7-e55f-4be6-b11f-00c3d4a2d701", 2020-07-17T12:17:01.581 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 3 2020-07-17T12:17:01.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:01.581 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:17:01.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:01.582 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:17:01.582 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:17:01.582 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:17:01.582 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988221.023402_317.1", 2020-07-17T12:17:01.583 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:17:01.583 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 80, 2020-07-17T12:17:01.583 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:17:01.583 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:01.023402Z", 2020-07-17T12:17:01.583 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 3 2020-07-17T12:17:01.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:01.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:01.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:01.584 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:17:01.584 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:17:01.584 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:17:01.585 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:17:01.585 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:17:01.585 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:17:01.585 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:17:01.585 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:17:01.586 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 3 2020-07-17T12:17:01.586 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:01.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:01.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:01.586 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:17:01.586 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:17:01.587 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:17:01.587 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:17:01.587 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:17:01.587 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:17:01.587 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:17:01.588 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:17:01.588 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 3 2020-07-17T12:17:01.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:01.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:01.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:01.589 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:17:01.589 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:17:01.589 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:17:01.589 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:17:01.589 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:17:01.590 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:17:01.590 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:17:01.590 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:17:01.590 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 3 2020-07-17T12:17:01.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:01.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:01.591 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:17:01.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:01.591 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:17:01.591 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 80, 2020-07-17T12:17:01.591 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 80 2020-07-17T12:17:01.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:01.592 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:17:01.592 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-07-17T12:17:01.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:17:01.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:17:01.594 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:17:01.660 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:17:01.664 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:17:01.688 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:17:01.680+0000 7f147fb630c0 0 NOTICE: overriding master zone: f2444b35-3e93-4fbd-99ef-52c98595b68f 2020-07-17T12:17:01.689 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:17:01.680+0000 7f147fb630c0 0 failed reading obj info from .rgw.root:zone_info.7326649b-420d-48b0-8ce4-28b020e4cf14: (2) No such file or directory 2020-07-17T12:17:01.690 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:17:01.680+0000 7f147fb630c0 0 WARNING: could not read zone params for zone id=7326649b-420d-48b0-8ce4-28b020e4cf14 name=test-zone4 2020-07-17T12:17:01.711 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:17:01.684+0000 7f147fb630c0 0 failed reading obj info from .rgw.root:zone_info.f2444b35-3e93-4fbd-99ef-52c98595b68f: (2) No such file or directory 2020-07-17T12:17:01.712 INFO:teuthology.orchestra.run.smithi146.stderr:2020-07-17T12:17:01.684+0000 7f147fb630c0 0 WARNING: could not read zone params for zone id=f2444b35-3e93-4fbd-99ef-52c98595b68f name=test-zone2 2020-07-17T12:17:01.712 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:17:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:17:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone1", 2020-07-17T12:17:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-07-17T12:17:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "control_pool": "test-zone1.rgw.control", 2020-07-17T12:17:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-07-17T12:17:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-07-17T12:17:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "log_pool": "test-zone1.rgw.log", 2020-07-17T12:17:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-07-17T12:17:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-07-17T12:17:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-07-17T12:17:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-07-17T12:17:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-07-17T12:17:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-07-17T12:17:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-07-17T12:17:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-07-17T12:17:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-07-17T12:17:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "system_key": { 2020-07-17T12:17:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:17:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:17:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_pools": [ 2020-07-17T12:17:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "default-placement", 2020-07-17T12:17:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:17:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-07-17T12:17:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": { 2020-07-17T12:17:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD": { 2020-07-17T12:17:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-07-17T12:17:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "compression_type": "random" 2020-07-17T12:17:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-07-17T12:17:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "index_type": 0 2020-07-17T12:17:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca" 2020-07-17T12:17:01.725 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:17:01.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:17:01.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:17:01.727 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:17:01.794 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:17:01.794 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:17:02.079 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:17:02.079 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:17:02.079 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 1, 2020-07-17T12:17:02.080 INFO:teuthology.orchestra.run.smithi146.stdout: "predecessor_uuid": "159da1b7-e55f-4be6-b11f-00c3d4a2d701", 2020-07-17T12:17:02.080 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": [ 2020-07-17T12:17:02.080 INFO:teuthology.orchestra.run.smithi146.stdout: "1_1594988221.023402_317.1", 2020-07-17T12:17:02.080 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2020-07-17T12:17:02.080 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2020-07-17T12:17:02.081 INFO:teuthology.orchestra.run.smithi146.stdout: "" 2020-07-17T12:17:02.081 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.081 INFO:teuthology.orchestra.run.smithi146.stdout: "period_map": { 2020-07-17T12:17:02.081 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:17:02.082 INFO:teuthology.orchestra.run.smithi146.stdout: "zonegroups": [ 2020-07-17T12:17:02.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.082 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "test-zonegroup", 2020-07-17T12:17:02.082 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zonegroup", 2020-07-17T12:17:02.082 INFO:teuthology.orchestra.run.smithi146.stdout: "api_name": "", 2020-07-17T12:17:02.083 INFO:teuthology.orchestra.run.smithi146.stdout: "is_master": "true", 2020-07-17T12:17:02.083 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:17:02.084 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:17:02.084 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.084 INFO:teuthology.orchestra.run.smithi146.stdout: "hostnames": [], 2020-07-17T12:17:02.084 INFO:teuthology.orchestra.run.smithi146.stdout: "hostnames_s3website": [], 2020-07-17T12:17:02.084 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:17:02.085 INFO:teuthology.orchestra.run.smithi146.stdout: "zones": [ 2020-07-17T12:17:02.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.085 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:17:02.085 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone3", 2020-07-17T12:17:02.085 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:17:02.086 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi146.front.sepia.ceph.com:8001" 2020-07-17T12:17:02.086 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.086 INFO:teuthology.orchestra.run.smithi146.stdout: "log_meta": "false", 2020-07-17T12:17:02.086 INFO:teuthology.orchestra.run.smithi146.stdout: "log_data": "true", 2020-07-17T12:17:02.086 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:17:02.087 INFO:teuthology.orchestra.run.smithi146.stdout: "read_only": "false", 2020-07-17T12:17:02.087 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_type": "", 2020-07-17T12:17:02.087 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from_all": "true", 2020-07-17T12:17:02.087 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from": [], 2020-07-17T12:17:02.087 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_zone": "" 2020-07-17T12:17:02.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:02.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.088 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:17:02.088 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone4", 2020-07-17T12:17:02.088 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:17:02.089 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi052.front.sepia.ceph.com:8001" 2020-07-17T12:17:02.089 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.089 INFO:teuthology.orchestra.run.smithi146.stdout: "log_meta": "false", 2020-07-17T12:17:02.089 INFO:teuthology.orchestra.run.smithi146.stdout: "log_data": "true", 2020-07-17T12:17:02.089 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:17:02.089 INFO:teuthology.orchestra.run.smithi146.stdout: "read_only": "false", 2020-07-17T12:17:02.090 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_type": "pubsub", 2020-07-17T12:17:02.090 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from_all": "false", 2020-07-17T12:17:02.090 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from": [ 2020-07-17T12:17:02.090 INFO:teuthology.orchestra.run.smithi146.stdout: "test-zone1" 2020-07-17T12:17:02.090 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_zone": "" 2020-07-17T12:17:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:17:02.092 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone1", 2020-07-17T12:17:02.092 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:17:02.092 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:17:02.092 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.092 INFO:teuthology.orchestra.run.smithi146.stdout: "log_meta": "false", 2020-07-17T12:17:02.093 INFO:teuthology.orchestra.run.smithi146.stdout: "log_data": "true", 2020-07-17T12:17:02.093 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:17:02.093 INFO:teuthology.orchestra.run.smithi146.stdout: "read_only": "false", 2020-07-17T12:17:02.093 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_type": "", 2020-07-17T12:17:02.094 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from_all": "true", 2020-07-17T12:17:02.094 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from": [], 2020-07-17T12:17:02.094 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_zone": "" 2020-07-17T12:17:02.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:02.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.095 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:17:02.095 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test-zone2", 2020-07-17T12:17:02.096 INFO:teuthology.orchestra.run.smithi146.stdout: "endpoints": [ 2020-07-17T12:17:02.096 INFO:teuthology.orchestra.run.smithi146.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2020-07-17T12:17:02.096 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.096 INFO:teuthology.orchestra.run.smithi146.stdout: "log_meta": "false", 2020-07-17T12:17:02.096 INFO:teuthology.orchestra.run.smithi146.stdout: "log_data": "true", 2020-07-17T12:17:02.096 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:17:02.097 INFO:teuthology.orchestra.run.smithi146.stdout: "read_only": "false", 2020-07-17T12:17:02.097 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_type": "", 2020-07-17T12:17:02.097 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from_all": "true", 2020-07-17T12:17:02.098 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_from": [], 2020-07-17T12:17:02.098 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_zone": "" 2020-07-17T12:17:02.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:02.099 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.099 INFO:teuthology.orchestra.run.smithi146.stdout: "placement_targets": [ 2020-07-17T12:17:02.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.099 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default-placement", 2020-07-17T12:17:02.099 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2020-07-17T12:17:02.100 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_classes": [ 2020-07-17T12:17:02.100 INFO:teuthology.orchestra.run.smithi146.stdout: "STANDARD" 2020-07-17T12:17:02.100 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:17:02.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:02.100 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.100 INFO:teuthology.orchestra.run.smithi146.stdout: "default_placement": "default-placement", 2020-07-17T12:17:02.101 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:17:02.101 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_policy": { 2020-07-17T12:17:02.101 INFO:teuthology.orchestra.run.smithi146.stdout: "groups": [] 2020-07-17T12:17:02.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:02.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:02.102 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2020-07-17T12:17:02.105 INFO:teuthology.orchestra.run.smithi146.stdout: "short_zone_ids": [ 2020-07-17T12:17:02.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.105 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:17:02.106 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 2627409927 2020-07-17T12:17:02.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:02.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.106 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:17:02.107 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 3854808555 2020-07-17T12:17:02.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:02.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.107 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:17:02.107 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 340514820 2020-07-17T12:17:02.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:02.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:17:02.108 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:17:02.108 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 3926357550 2020-07-17T12:17:02.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:02.109 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:17:02.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:02.109 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zonegroup": "test-zonegroup", 2020-07-17T12:17:02.110 INFO:teuthology.orchestra.run.smithi146.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:17:02.110 INFO:teuthology.orchestra.run.smithi146.stdout: "period_config": { 2020-07-17T12:17:02.110 INFO:teuthology.orchestra.run.smithi146.stdout: "bucket_quota": { 2020-07-17T12:17:02.110 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:17:02.110 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:17:02.111 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:17:02.111 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:17:02.111 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:17:02.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:02.111 INFO:teuthology.orchestra.run.smithi146.stdout: "user_quota": { 2020-07-17T12:17:02.112 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": false, 2020-07-17T12:17:02.112 INFO:teuthology.orchestra.run.smithi146.stdout: "check_on_raw": false, 2020-07-17T12:17:02.112 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size": -1, 2020-07-17T12:17:02.112 INFO:teuthology.orchestra.run.smithi146.stdout: "max_size_kb": 0, 2020-07-17T12:17:02.112 INFO:teuthology.orchestra.run.smithi146.stdout: "max_objects": -1 2020-07-17T12:17:02.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:17:02.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:17:02.113 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:17:02.113 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_name": "test-realm", 2020-07-17T12:17:02.113 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:17:02.114 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:17:02.115 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-07-17T12:18:02.117 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-44 2020-07-17T12:18:02.167 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-45 2020-07-17T12:18:02.237 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-07-17T12:18:02.238 INFO:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-07-17T12:18:02.242 INFO:tasks.rgw.c1.client.1:Started 2020-07-17T12:18:02.243 INFO:teuthology.orchestra.run.smithi146:> curl http://smithi146.front.sepia.ceph.com:8001 2020-07-17T12:18:02.259 INFO:teuthology.orchestra.run.smithi146.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:18:02.259 INFO:teuthology.orchestra.run.smithi146.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:18:02.262 INFO:teuthology.orchestra.run.smithi146.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi146.front.sepia.ceph.com port 8001: Connection refused 2020-07-17T12:18:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-17T12:18:03.266 INFO:teuthology.orchestra.run.smithi146:> curl http://smithi146.front.sepia.ceph.com:8001 2020-07-17T12:18:03.282 INFO:teuthology.orchestra.run.smithi146.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-17T12:18:03.282 INFO:teuthology.orchestra.run.smithi146.stderr: Dload Upload Total Spent Left Speed 2020-07-17T12:18:03.287 INFO:teuthology.orchestra.run.smithi146.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-17T12:18:03.288 INFO:teuthology.orchestra.run.smithi146.stdout:anonymous 2020-07-17T12:18:03.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:03.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:03.290 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:03.369 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:03.369 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:03.544 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=927461a0-cbf0-4f44-ab5d-2bc9be48438c 2020-07-17T12:18:03.577 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:03.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:03.578 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988282.203742_304.1", 2020-07-17T12:18:03.578 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:18:02.203742Z" 2020-07-17T12:18:03.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:03.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:03.579 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:18:03.579 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:18:03.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:03.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:03.580 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:18:03.580 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:18:03.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:03.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:03.580 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:18:03.581 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:18:03.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:03.581 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:03.582 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:18:03.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:03.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:03.584 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:03.625 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:03.626 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:03.881 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:18:03.881 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:18:03.881 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:18:03.882 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:18:03.882 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:18:03.882 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:18:03.882 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:18:03.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:03.882 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:18:03.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:03.883 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:18:03.883 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:03.883 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:18:03.883 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988282.203742_304.1", 2020-07-17T12:18:03.884 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:03.884 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:18:03.884 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:03.884 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.203742Z", 2020-07-17T12:18:03.884 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:18:03.885 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:03.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:03.885 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:03.885 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:18:03.885 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:03.885 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:18:03.886 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:18:03.886 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:03.886 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:03.886 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:03.886 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:18:03.887 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:18:03.887 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:03.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:03.887 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:03.887 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:18:03.888 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:03.888 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:18:03.888 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:18:03.888 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:03.888 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:03.888 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:03.889 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:18:03.889 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:18:03.889 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:03.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:03.889 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:03.890 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:18:03.890 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:03.890 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:18:03.890 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:18:03.890 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:03.891 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:03.891 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:03.891 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:18:03.891 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:18:03.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:03.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:18:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:18:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:18:03.892 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:18:03.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:03.893 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:18:03.894 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:18:03.895 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:18:03.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:03.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:03.896 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:03.938 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:03.938 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:04.165 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:18:04.165 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:18:04.166 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:18:04.166 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:18:04.166 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:18:04.166 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:18:04.167 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:18:04.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:04.167 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:18:04.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:04.167 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:18:04.167 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:04.168 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:18:04.168 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988282.203742_304.1", 2020-07-17T12:18:04.168 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:04.168 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:18:04.168 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:04.169 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.203742Z", 2020-07-17T12:18:04.169 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:18:04.169 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:04.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:04.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:04.170 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:18:04.170 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:04.170 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:18:04.170 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:18:04.170 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:04.171 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:04.171 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:04.171 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:18:04.171 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:18:04.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:04.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:04.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:04.173 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:18:04.173 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:04.173 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:18:04.174 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:18:04.174 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:04.174 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:04.174 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:04.175 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:18:04.175 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:18:04.176 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:04.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:04.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:04.177 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:18:04.177 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:04.177 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:18:04.180 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:18:04.180 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:04.180 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:04.180 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:04.181 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:18:04.181 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:18:04.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:04.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:04.182 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:18:04.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:04.182 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:18:04.183 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:18:04.183 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:18:04.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:04.184 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:18:04.185 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:18:04.186 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:18:04.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:04.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:04.187 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:04.227 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:04.227 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:04.506 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:18:04.506 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:18:04.507 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:18:04.507 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:18:04.507 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:18:04.508 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:18:04.508 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:18:04.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:04.509 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:18:04.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:04.509 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:18:04.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:04.510 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:18:04.510 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988282.203742_304.1", 2020-07-17T12:18:04.510 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:04.511 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:18:04.511 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:04.511 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.203742Z", 2020-07-17T12:18:04.512 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:18:04.512 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:04.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:04.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:04.513 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:18:04.513 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:04.514 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:18:04.514 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:18:04.514 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:04.515 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:04.515 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:04.515 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:18:04.515 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:18:04.515 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:04.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:04.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:04.516 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:18:04.516 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:04.516 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:18:04.517 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:18:04.517 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:04.517 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:04.517 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:04.517 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:18:04.518 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:18:04.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:04.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:04.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:04.518 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:18:04.519 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:04.519 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:18:04.519 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:18:04.519 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:04.519 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:04.520 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:04.520 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:18:04.520 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:18:04.520 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:04.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:04.521 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:18:04.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:04.521 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:18:04.521 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:18:04.522 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:18:04.522 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:04.522 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:18:04.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-17T12:18:04.524 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-39 zones={test-zone2, test-zone1} 2020-07-17T12:18:04.545 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:04.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:04.662 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-39 zones={test-zone2, test-zone1} 2020-07-17T12:18:04.663 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-39 zones={test-zone3, test-zone1} 2020-07-17T12:18:04.680 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:04.730 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:04.792 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-39 zones={test-zone3, test-zone1} 2020-07-17T12:18:04.793 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-39 zones={test-zone1, test-zone2} 2020-07-17T12:18:04.805 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:04.857 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:04.925 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-39 zones={test-zone1, test-zone2} 2020-07-17T12:18:04.926 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-39 zones={test-zone3, test-zone2} 2020-07-17T12:18:04.938 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:04.990 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:05.056 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-39 zones={test-zone3, test-zone2} 2020-07-17T12:18:05.057 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-39 zones={test-zone1, test-zone3} 2020-07-17T12:18:05.070 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:05.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:05.183 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-39 zones={test-zone1, test-zone3} 2020-07-17T12:18:05.184 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-39 zones={test-zone2, test-zone3} 2020-07-17T12:18:05.195 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:05.242 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:05.304 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-39 zones={test-zone2, test-zone3} 2020-07-17T12:18:05.304 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-40 zones={test-zone2, test-zone1} 2020-07-17T12:18:05.320 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:05.370 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:05.433 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-40 zones={test-zone2, test-zone1} 2020-07-17T12:18:05.433 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-40 zones={test-zone3, test-zone1} 2020-07-17T12:18:05.445 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:05.494 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:05.556 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-40 zones={test-zone3, test-zone1} 2020-07-17T12:18:05.557 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-40 zones={test-zone1, test-zone2} 2020-07-17T12:18:05.568 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:05.614 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:05.685 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-40 zones={test-zone1, test-zone2} 2020-07-17T12:18:05.685 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-40 zones={test-zone3, test-zone2} 2020-07-17T12:18:05.697 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:05.746 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:05.810 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-40 zones={test-zone3, test-zone2} 2020-07-17T12:18:05.810 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-40 zones={test-zone1, test-zone3} 2020-07-17T12:18:05.823 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:05.870 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:05.934 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-40 zones={test-zone1, test-zone3} 2020-07-17T12:18:05.935 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-40 zones={test-zone2, test-zone3} 2020-07-17T12:18:05.945 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:05.994 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:06.057 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-40 zones={test-zone2, test-zone3} 2020-07-17T12:18:06.058 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-41 zones={test-zone2, test-zone1} 2020-07-17T12:18:06.081 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:06.130 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:06.200 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-41 zones={test-zone2, test-zone1} 2020-07-17T12:18:06.200 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-41 zones={test-zone3, test-zone1} 2020-07-17T12:18:06.213 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:06.266 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:06.329 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-41 zones={test-zone3, test-zone1} 2020-07-17T12:18:06.330 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-41 zones={test-zone1, test-zone2} 2020-07-17T12:18:06.341 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:06.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:06.454 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-41 zones={test-zone1, test-zone2} 2020-07-17T12:18:06.455 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-41 zones={test-zone3, test-zone2} 2020-07-17T12:18:06.464 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:06.518 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:06.581 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-41 zones={test-zone3, test-zone2} 2020-07-17T12:18:06.582 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-41 zones={test-zone1, test-zone3} 2020-07-17T12:18:06.595 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:06.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:06.705 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-41 zones={test-zone1, test-zone3} 2020-07-17T12:18:06.706 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-41 zones={test-zone2, test-zone3} 2020-07-17T12:18:06.717 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:06.766 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:06.830 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-41 zones={test-zone2, test-zone3} 2020-07-17T12:18:06.831 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-42 zones={test-zone2, test-zone1} 2020-07-17T12:18:06.842 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:06.890 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:06.957 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-42 zones={test-zone2, test-zone1} 2020-07-17T12:18:06.958 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-42 zones={test-zone3, test-zone1} 2020-07-17T12:18:06.967 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:07.014 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:07.081 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-42 zones={test-zone3, test-zone1} 2020-07-17T12:18:07.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-42 zones={test-zone1, test-zone2} 2020-07-17T12:18:07.094 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:07.146 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:07.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-42 zones={test-zone1, test-zone2} 2020-07-17T12:18:07.210 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-42 zones={test-zone3, test-zone2} 2020-07-17T12:18:07.220 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:07.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:07.347 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-42 zones={test-zone3, test-zone2} 2020-07-17T12:18:07.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-42 zones={test-zone1, test-zone3} 2020-07-17T12:18:07.359 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:07.406 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:07.467 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-42 zones={test-zone1, test-zone3} 2020-07-17T12:18:07.468 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-42 zones={test-zone2, test-zone3} 2020-07-17T12:18:07.479 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:07.526 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:07.590 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-42 zones={test-zone2, test-zone3} 2020-07-17T12:18:07.590 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-43 zones={test-zone2, test-zone1} 2020-07-17T12:18:07.600 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:07.655 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:07.721 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-43 zones={test-zone2, test-zone1} 2020-07-17T12:18:07.721 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-43 zones={test-zone3, test-zone1} 2020-07-17T12:18:07.731 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:07.778 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:07.840 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-43 zones={test-zone3, test-zone1} 2020-07-17T12:18:07.841 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-43 zones={test-zone1, test-zone2} 2020-07-17T12:18:07.853 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:07.918 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:08.014 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-43 zones={test-zone1, test-zone2} 2020-07-17T12:18:08.015 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-43 zones={test-zone3, test-zone2} 2020-07-17T12:18:08.040 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:08.108 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:08.203 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-43 zones={test-zone3, test-zone2} 2020-07-17T12:18:08.204 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-43 zones={test-zone1, test-zone3} 2020-07-17T12:18:08.224 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:08.278 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:08.356 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-43 zones={test-zone1, test-zone3} 2020-07-17T12:18:08.357 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-43 zones={test-zone2, test-zone3} 2020-07-17T12:18:08.375 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:08.428 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:08.500 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-43 zones={test-zone2, test-zone3} 2020-07-17T12:18:08.501 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-44 zones={test-zone2, test-zone1} 2020-07-17T12:18:08.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:08.567 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:08.634 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-44 zones={test-zone2, test-zone1} 2020-07-17T12:18:08.635 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-44 zones={test-zone3, test-zone1} 2020-07-17T12:18:08.647 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:08.694 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:08.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-44 zones={test-zone3, test-zone1} 2020-07-17T12:18:08.763 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-44 zones={test-zone1, test-zone2} 2020-07-17T12:18:08.774 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:08.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:08.886 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-44 zones={test-zone1, test-zone2} 2020-07-17T12:18:08.887 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-44 zones={test-zone3, test-zone2} 2020-07-17T12:18:08.900 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:08.951 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:09.013 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-44 zones={test-zone3, test-zone2} 2020-07-17T12:18:09.014 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-44 zones={test-zone1, test-zone3} 2020-07-17T12:18:09.028 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:09.078 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:09.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-44 zones={test-zone1, test-zone3} 2020-07-17T12:18:09.155 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-44 zones={test-zone2, test-zone3} 2020-07-17T12:18:09.165 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:09.214 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:09.282 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-44 zones={test-zone2, test-zone3} 2020-07-17T12:18:09.282 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-45 zones={test-zone2, test-zone1} 2020-07-17T12:18:09.295 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:09.342 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:09.405 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-45 zones={test-zone2, test-zone1} 2020-07-17T12:18:09.406 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-45 zones={test-zone3, test-zone1} 2020-07-17T12:18:09.414 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:09.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:09.529 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-45 zones={test-zone3, test-zone1} 2020-07-17T12:18:09.529 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-45 zones={test-zone1, test-zone2} 2020-07-17T12:18:09.554 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:09.602 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:09.665 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-45 zones={test-zone1, test-zone2} 2020-07-17T12:18:09.666 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-45 zones={test-zone3, test-zone2} 2020-07-17T12:18:09.676 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:09.722 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:09.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-45 zones={test-zone3, test-zone2} 2020-07-17T12:18:09.790 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-45 zones={test-zone1, test-zone3} 2020-07-17T12:18:09.803 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:09.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:09.912 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-45 zones={test-zone1, test-zone3} 2020-07-17T12:18:09.913 INFO:tasks.rgw_multi.zone_rados:comparing bucket=lbjidp-45 zones={test-zone2, test-zone3} 2020-07-17T12:18:09.935 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-17T12:18:09.982 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-17T12:18:10.048 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=lbjidp-45 zones={test-zone2, test-zone3} 2020-07-17T12:18:10.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:10.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:10.050 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 34954458-202e-48c3-864b-c00b15f41dd5 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:10.083 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:10.084 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:10.266 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:10.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.267 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987471.882144_44.1", 2020-07-17T12:18:10.267 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "user", 2020-07-17T12:18:10.267 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rgw-multisite-test-user", 2020-07-17T12:18:10.267 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.882144008Z", 2020-07-17T12:18:10.268 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.268 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.268 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.268 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.269 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.269 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_uJ2kpFkaIMR3kSeSIGfEB3T", 2020-07-17T12:18:10.269 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.269 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.270 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.271 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987471.890977_45.1", 2020-07-17T12:18:10.271 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "user", 2020-07-17T12:18:10.271 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rgw-multisite-test-user", 2020-07-17T12:18:10.271 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.890977275Z", 2020-07-17T12:18:10.271 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.272 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.272 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.272 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.272 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.273 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_uJ2kpFkaIMR3kSeSIGfEB3T", 2020-07-17T12:18:10.273 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.273 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.273 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.275 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.172466_48.1", 2020-07-17T12:18:10.275 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.275 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-10:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.10", 2020-07-17T12:18:10.275 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.172466793Z", 2020-07-17T12:18:10.275 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.276 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.276 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.276 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.276 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.277 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_mG7u6kOIVSSgWYLweK5uzvb", 2020-07-17T12:18:10.277 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.277 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.277 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.278 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.278 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.278 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.177556_49.1", 2020-07-17T12:18:10.278 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.279 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-10:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.10", 2020-07-17T12:18:10.279 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.177556387Z", 2020-07-17T12:18:10.279 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.279 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.279 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.280 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.280 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.280 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_mG7u6kOIVSSgWYLweK5uzvb", 2020-07-17T12:18:10.280 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.281 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.281 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.282 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.182179_50.1", 2020-07-17T12:18:10.282 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.282 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-10", 2020-07-17T12:18:10.283 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.182179376Z", 2020-07-17T12:18:10.283 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.283 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.283 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.284 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.284 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.286 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_yuMwADwMvlFMi5Y6H00iObr", 2020-07-17T12:18:10.286 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.286 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.286 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.287 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.186476_51.1", 2020-07-17T12:18:10.288 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.288 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-10", 2020-07-17T12:18:10.288 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.186476843Z", 2020-07-17T12:18:10.288 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.288 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.289 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.289 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.289 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.289 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_yuMwADwMvlFMi5Y6H00iObr", 2020-07-17T12:18:10.290 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.290 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.290 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.291 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.216713_52.1", 2020-07-17T12:18:10.291 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.291 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-11:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.11", 2020-07-17T12:18:10.292 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.216713690Z", 2020-07-17T12:18:10.292 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.292 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.292 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.292 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.293 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.293 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gq5MHLpwjP0npYemQW4217g", 2020-07-17T12:18:10.293 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.294 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.294 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.295 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.222092_53.1", 2020-07-17T12:18:10.295 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.295 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-11:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.11", 2020-07-17T12:18:10.295 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.222092496Z", 2020-07-17T12:18:10.295 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.296 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.296 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.296 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.296 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.297 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gq5MHLpwjP0npYemQW4217g", 2020-07-17T12:18:10.297 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.297 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.300 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.302 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.223732_54.1", 2020-07-17T12:18:10.302 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.302 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-11", 2020-07-17T12:18:10.302 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.223732073Z", 2020-07-17T12:18:10.302 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.302 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.303 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.303 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.303 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.303 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_hmDQZxZXEQLZ3GX5-Ia1DXm", 2020-07-17T12:18:10.304 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.304 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.304 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.305 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.305 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.227773_55.1", 2020-07-17T12:18:10.305 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.306 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-11", 2020-07-17T12:18:10.306 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.227773848Z", 2020-07-17T12:18:10.306 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.306 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.306 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.307 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.307 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.307 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_hmDQZxZXEQLZ3GX5-Ia1DXm", 2020-07-17T12:18:10.307 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.308 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.308 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.309 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.263759_56.1", 2020-07-17T12:18:10.309 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.309 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-12:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.12", 2020-07-17T12:18:10.309 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.263759779Z", 2020-07-17T12:18:10.310 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.310 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.310 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.310 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.311 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_kJPn3awfkoAAv2YoILjX1YX", 2020-07-17T12:18:10.311 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.311 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.312 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.312 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.272286_57.1", 2020-07-17T12:18:10.313 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.313 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-12:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.12", 2020-07-17T12:18:10.313 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.272286467Z", 2020-07-17T12:18:10.313 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.313 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.314 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.314 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.314 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.314 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_kJPn3awfkoAAv2YoILjX1YX", 2020-07-17T12:18:10.315 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.315 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.315 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.316 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.274467_58.1", 2020-07-17T12:18:10.316 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.316 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-12", 2020-07-17T12:18:10.317 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.274467332Z", 2020-07-17T12:18:10.317 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.317 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.317 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.317 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.318 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.318 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_WFmqiEkT2ufoxFeioom356B", 2020-07-17T12:18:10.318 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.319 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.319 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.320 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.278144_59.1", 2020-07-17T12:18:10.320 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.320 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-12", 2020-07-17T12:18:10.321 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.278144599Z", 2020-07-17T12:18:10.321 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.321 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.321 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.321 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.322 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.322 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_WFmqiEkT2ufoxFeioom356B", 2020-07-17T12:18:10.322 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.323 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.323 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.323 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.323 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.324 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.712640_60.1", 2020-07-17T12:18:10.324 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.324 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-13:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.13", 2020-07-17T12:18:10.324 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.712640382Z", 2020-07-17T12:18:10.325 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.325 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.325 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.325 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.326 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.326 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_coVsWkNqvKBUCWs8AacIoNl", 2020-07-17T12:18:10.326 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.326 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.329 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.329 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.331 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.717263_61.1", 2020-07-17T12:18:10.331 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.332 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-13:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.13", 2020-07-17T12:18:10.332 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.717263960Z", 2020-07-17T12:18:10.332 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.332 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.332 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.333 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.333 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_coVsWkNqvKBUCWs8AacIoNl", 2020-07-17T12:18:10.333 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.334 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.334 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.335 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.719823_62.1", 2020-07-17T12:18:10.335 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.335 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-13", 2020-07-17T12:18:10.336 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.719823082Z", 2020-07-17T12:18:10.336 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.336 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.336 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.336 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.337 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.337 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dWoVeSt6MXyutBS6iBOB_SN", 2020-07-17T12:18:10.337 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.338 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.338 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.339 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.725332_63.1", 2020-07-17T12:18:10.339 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.339 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-13", 2020-07-17T12:18:10.339 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.725332531Z", 2020-07-17T12:18:10.339 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.340 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.340 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.340 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.340 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dWoVeSt6MXyutBS6iBOB_SN", 2020-07-17T12:18:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.342 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.756675_64.1", 2020-07-17T12:18:10.343 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.343 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-14:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.14", 2020-07-17T12:18:10.343 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.756675726Z", 2020-07-17T12:18:10.343 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.343 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.343 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_BWPfn12CXl2TlU4AWhdAySv", 2020-07-17T12:18:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.761544_65.1", 2020-07-17T12:18:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-14:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.14", 2020-07-17T12:18:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.761544051Z", 2020-07-17T12:18:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_BWPfn12CXl2TlU4AWhdAySv", 2020-07-17T12:18:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.763526_66.1", 2020-07-17T12:18:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-14", 2020-07-17T12:18:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.763526945Z", 2020-07-17T12:18:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_irk8P5j0CkMCoEl6rP2eANQ", 2020-07-17T12:18:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.767394_67.1", 2020-07-17T12:18:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-14", 2020-07-17T12:18:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.767394556Z", 2020-07-17T12:18:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_irk8P5j0CkMCoEl6rP2eANQ", 2020-07-17T12:18:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.810791_68.1", 2020-07-17T12:18:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-15:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.15", 2020-07-17T12:18:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.810791998Z", 2020-07-17T12:18:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_aYP5BWVqE2ZQzg8KQUMoKIk", 2020-07-17T12:18:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.816286_69.1", 2020-07-17T12:18:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-15:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.15", 2020-07-17T12:18:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.816286566Z", 2020-07-17T12:18:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_aYP5BWVqE2ZQzg8KQUMoKIk", 2020-07-17T12:18:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.820905_70.1", 2020-07-17T12:18:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-15", 2020-07-17T12:18:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.820905407Z", 2020-07-17T12:18:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_rpYWMBKFNIdTSlnC72SKitx", 2020-07-17T12:18:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.830894_71.1", 2020-07-17T12:18:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-15", 2020-07-17T12:18:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.830894206Z", 2020-07-17T12:18:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_rpYWMBKFNIdTSlnC72SKitx", 2020-07-17T12:18:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.232376_72.1", 2020-07-17T12:18:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.232376434Z", 2020-07-17T12:18:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.236895_73.1", 2020-07-17T12:18:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.236895263Z", 2020-07-17T12:18:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.238681_74.1", 2020-07-17T12:18:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16", 2020-07-17T12:18:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.238681635Z", 2020-07-17T12:18:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_lEb4NYqynMKhYQrCh3ur9Pg", 2020-07-17T12:18:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.242366_75.1", 2020-07-17T12:18:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16", 2020-07-17T12:18:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.242366508Z", 2020-07-17T12:18:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_lEb4NYqynMKhYQrCh3ur9Pg", 2020-07-17T12:18:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.269805_76.1", 2020-07-17T12:18:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.269805039Z", 2020-07-17T12:18:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.273253_77.1", 2020-07-17T12:18:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.273253052Z", 2020-07-17T12:18:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.274824_78.1", 2020-07-17T12:18:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17", 2020-07-17T12:18:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.274824384Z", 2020-07-17T12:18:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_kC34LrBQXhenfmJdKuMdgnY", 2020-07-17T12:18:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.277982_79.1", 2020-07-17T12:18:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17", 2020-07-17T12:18:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.277982524Z", 2020-07-17T12:18:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_kC34LrBQXhenfmJdKuMdgnY", 2020-07-17T12:18:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.321344_80.1", 2020-07-17T12:18:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.321344326Z", 2020-07-17T12:18:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.325975_81.1", 2020-07-17T12:18:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.325975985Z", 2020-07-17T12:18:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.327813_82.1", 2020-07-17T12:18:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18", 2020-07-17T12:18:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.327813080Z", 2020-07-17T12:18:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_0c-EMQmUZNyncsOLFYYvBXZ", 2020-07-17T12:18:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.331302_83.1", 2020-07-17T12:18:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18", 2020-07-17T12:18:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.331302239Z", 2020-07-17T12:18:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_0c-EMQmUZNyncsOLFYYvBXZ", 2020-07-17T12:18:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.349661_84.1", 2020-07-17T12:18:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.349661555Z", 2020-07-17T12:18:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.354642_85.1", 2020-07-17T12:18:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.354642695Z", 2020-07-17T12:18:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.358809_86.1", 2020-07-17T12:18:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.358809552Z", 2020-07-17T12:18:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.364350_87.1", 2020-07-17T12:18:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.364350896Z", 2020-07-17T12:18:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.374842_88.1", 2020-07-17T12:18:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.374842199Z", 2020-07-17T12:18:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.381436_89.1", 2020-07-17T12:18:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.381436296Z", 2020-07-17T12:18:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.580189_90.1", 2020-07-17T12:18:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.580189016Z", 2020-07-17T12:18:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.583928_91.1", 2020-07-17T12:18:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.583928438Z", 2020-07-17T12:18:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.585595_92.1", 2020-07-17T12:18:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19", 2020-07-17T12:18:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.585595449Z", 2020-07-17T12:18:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Y0anvog_R4e6DQW9utotNY_", 2020-07-17T12:18:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.589056_93.1", 2020-07-17T12:18:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19", 2020-07-17T12:18:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.589056210Z", 2020-07-17T12:18:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Y0anvog_R4e6DQW9utotNY_", 2020-07-17T12:18:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.596888_94.1", 2020-07-17T12:18:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.596888308Z", 2020-07-17T12:18:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.602473_95.1", 2020-07-17T12:18:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.602473621Z", 2020-07-17T12:18:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987998.537142_96.1", 2020-07-17T12:18:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:18.537142107Z", 2020-07-17T12:18:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.469 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987998.541926_97.1", 2020-07-17T12:18:10.469 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.469 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:10.469 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:18.541926282Z", 2020-07-17T12:18:10.470 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.470 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.470 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.470 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.471 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.471 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.471 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.471 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.472 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.474 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987998.544300_98.1", 2020-07-17T12:18:10.474 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.475 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20", 2020-07-17T12:18:10.475 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:18.544300322Z", 2020-07-17T12:18:10.475 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.475 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.475 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.476 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.476 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.476 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_qoKh-ps-xZklh5sIuAPFkjZ", 2020-07-17T12:18:10.476 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.477 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.477 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.477 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.477 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.478 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987998.548993_99.1", 2020-07-17T12:18:10.478 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.478 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20", 2020-07-17T12:18:10.478 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:18.548993097Z", 2020-07-17T12:18:10.479 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.479 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.479 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.479 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.480 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.480 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_qoKh-ps-xZklh5sIuAPFkjZ", 2020-07-17T12:18:10.480 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.480 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.481 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.481 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.482 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988004.926158_100.1", 2020-07-17T12:18:10.482 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.482 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:10.482 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:24.926158716Z", 2020-07-17T12:18:10.482 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.483 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.483 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.483 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.483 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.484 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.484 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.484 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.484 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.485 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988004.936521_101.1", 2020-07-17T12:18:10.486 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.486 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:10.486 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:24.936521959Z", 2020-07-17T12:18:10.486 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.486 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.487 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.487 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.487 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.487 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.487 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.488 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.488 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.489 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988065.046888_102.1", 2020-07-17T12:18:10.489 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.489 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:10.490 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:14:25.046888654Z", 2020-07-17T12:18:10.490 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.490 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.490 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.490 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.491 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.491 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.491 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:10.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.491 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.492 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.493 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988065.056306_103.1", 2020-07-17T12:18:10.493 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.493 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:10.493 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:14:25.056306412Z", 2020-07-17T12:18:10.493 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.494 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.494 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.494 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.494 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.494 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:10.495 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:10.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.495 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.495 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.496 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.108561_114.1", 2020-07-17T12:18:10.496 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.497 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:10.497 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.108561553Z", 2020-07-17T12:18:10.497 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.497 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.497 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.497 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.498 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.498 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:10.498 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.499 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.499 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.500 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.112561_115.1", 2020-07-17T12:18:10.500 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.500 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:10.500 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.112561600Z", 2020-07-17T12:18:10.500 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.501 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.501 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.501 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.501 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.502 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:10.502 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.502 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.502 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.502 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.503 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.114341_116.1", 2020-07-17T12:18:10.503 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.504 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21", 2020-07-17T12:18:10.504 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.114341527Z", 2020-07-17T12:18:10.504 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.504 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.504 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.504 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.505 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.505 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_7IHshvdvzhTN_iaS63UlhhP", 2020-07-17T12:18:10.505 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.506 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.506 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.507 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.118095_117.1", 2020-07-17T12:18:10.507 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.507 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21", 2020-07-17T12:18:10.507 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.118095935Z", 2020-07-17T12:18:10.507 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.508 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.508 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.508 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.508 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.508 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_7IHshvdvzhTN_iaS63UlhhP", 2020-07-17T12:18:10.509 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.509 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.509 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.509 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.510 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.144536_119.1", 2020-07-17T12:18:10.510 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.511 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:10.511 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.144536481Z", 2020-07-17T12:18:10.511 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.511 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.511 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.511 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.512 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.512 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:10.512 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.513 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.513 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.514 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.148697_120.1", 2020-07-17T12:18:10.514 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.514 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:10.514 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.148697190Z", 2020-07-17T12:18:10.514 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.515 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.515 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.515 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.515 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.515 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:10.516 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.516 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.516 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.517 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.150760_121.1", 2020-07-17T12:18:10.517 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.517 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22", 2020-07-17T12:18:10.518 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.150760146Z", 2020-07-17T12:18:10.518 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.518 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.518 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.518 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.519 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.519 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gsPtnlqFRKAZ04Ktvzs95UN", 2020-07-17T12:18:10.519 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.519 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.520 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.521 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.155597_122.1", 2020-07-17T12:18:10.521 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.521 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22", 2020-07-17T12:18:10.521 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.155597648Z", 2020-07-17T12:18:10.521 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.521 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.522 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.522 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.522 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.522 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gsPtnlqFRKAZ04Ktvzs95UN", 2020-07-17T12:18:10.523 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.523 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.523 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.524 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.195741_123.1", 2020-07-17T12:18:10.524 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.524 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:10.525 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.195741697Z", 2020-07-17T12:18:10.525 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.525 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.525 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.525 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.526 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.526 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:10.526 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.527 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.527 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.528 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.200019_124.1", 2020-07-17T12:18:10.528 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.528 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:10.528 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.200019763Z", 2020-07-17T12:18:10.529 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.529 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.529 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.529 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.530 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.530 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:10.530 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.530 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.530 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.531 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.201967_125.1", 2020-07-17T12:18:10.532 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.532 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23", 2020-07-17T12:18:10.532 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.201967975Z", 2020-07-17T12:18:10.532 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.532 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.532 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.533 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.533 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.533 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_4EWRUD1pbHkY-FiDIwWfNaQ", 2020-07-17T12:18:10.533 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.534 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.534 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.535 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.205751_126.1", 2020-07-17T12:18:10.535 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.535 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23", 2020-07-17T12:18:10.535 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.205751555Z", 2020-07-17T12:18:10.536 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.536 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.536 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.536 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.536 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.537 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_4EWRUD1pbHkY-FiDIwWfNaQ", 2020-07-17T12:18:10.537 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.537 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.537 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.538 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.226616_127.1", 2020-07-17T12:18:10.538 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.539 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:10.539 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.226616279Z", 2020-07-17T12:18:10.539 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.539 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.539 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:10.540 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.540 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.540 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:10.540 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.541 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.541 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.542 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.231665_128.1", 2020-07-17T12:18:10.542 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.542 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:10.542 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.231665701Z", 2020-07-17T12:18:10.543 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.543 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.543 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:10.543 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.543 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.544 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:10.544 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.544 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.544 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.545 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.545 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.545 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.236032_130.1", 2020-07-17T12:18:10.545 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.546 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:10.546 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.236032642Z", 2020-07-17T12:18:10.546 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.546 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.546 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:10.547 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.547 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.547 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:10.547 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.548 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.548 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.548 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.548 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.549 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.240206_131.1", 2020-07-17T12:18:10.549 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.549 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:10.549 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.240206565Z", 2020-07-17T12:18:10.549 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.550 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.550 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:10.550 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.550 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.551 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:10.551 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.551 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.551 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.551 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.552 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.251296_132.1", 2020-07-17T12:18:10.552 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.553 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:10.553 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.251296772Z", 2020-07-17T12:18:10.553 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.553 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.554 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:10.554 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.554 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.554 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:10.555 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.555 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.555 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.556 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.262832_133.1", 2020-07-17T12:18:10.556 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.557 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:10.557 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.262832933Z", 2020-07-17T12:18:10.557 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.557 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.557 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:10.558 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.558 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.558 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:10.558 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.559 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.559 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.560 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.216266_162.1", 2020-07-17T12:18:10.560 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.560 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:10.560 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.216266874Z", 2020-07-17T12:18:10.561 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.561 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.561 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.561 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.562 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.562 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.562 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.562 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.562 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.563 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.220314_163.1", 2020-07-17T12:18:10.564 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.564 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:10.564 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.220314597Z", 2020-07-17T12:18:10.564 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.564 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.565 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.565 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.565 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.565 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.566 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.566 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.566 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.566 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.567 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.222492_164.1", 2020-07-17T12:18:10.567 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.567 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24", 2020-07-17T12:18:10.568 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.222492473Z", 2020-07-17T12:18:10.568 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.568 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.568 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.568 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.569 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.569 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1aTUtnO5v7_BzTHJ7dcWax9", 2020-07-17T12:18:10.569 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.569 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.570 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.578 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.226266_165.1", 2020-07-17T12:18:10.579 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.579 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24", 2020-07-17T12:18:10.579 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.226266565Z", 2020-07-17T12:18:10.579 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.579 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.580 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.580 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.580 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.580 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1aTUtnO5v7_BzTHJ7dcWax9", 2020-07-17T12:18:10.580 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.581 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.581 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.582 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.254586_166.1", 2020-07-17T12:18:10.582 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.582 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:10.582 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.254586031Z", 2020-07-17T12:18:10.583 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.583 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.583 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.583 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.584 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.584 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.584 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.584 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.584 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.585 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.259320_167.1", 2020-07-17T12:18:10.586 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.586 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:10.586 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.259320320Z", 2020-07-17T12:18:10.586 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.586 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.587 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.587 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.587 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.587 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.587 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.588 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.588 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.589 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.261985_168.1", 2020-07-17T12:18:10.589 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.589 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25", 2020-07-17T12:18:10.589 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.261985157Z", 2020-07-17T12:18:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.591 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_CJpPPaq5s3R_DLWK7_Dw6N9", 2020-07-17T12:18:10.591 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.591 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.591 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.592 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.268842_169.1", 2020-07-17T12:18:10.592 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.593 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25", 2020-07-17T12:18:10.593 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.268842944Z", 2020-07-17T12:18:10.593 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.593 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.593 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.594 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.594 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.594 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_CJpPPaq5s3R_DLWK7_Dw6N9", 2020-07-17T12:18:10.594 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.595 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.595 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.596 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.310784_170.1", 2020-07-17T12:18:10.596 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.596 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:10.596 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.310784639Z", 2020-07-17T12:18:10.596 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.597 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.598 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.319433_171.1", 2020-07-17T12:18:10.599 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.319433996Z", 2020-07-17T12:18:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.600 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.322090_172.1", 2020-07-17T12:18:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26", 2020-07-17T12:18:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.322090964Z", 2020-07-17T12:18:10.603 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.604 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_OI4YC_rV8-IvJU_ONTZMZ22", 2020-07-17T12:18:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.605 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.326100_173.1", 2020-07-17T12:18:10.606 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26", 2020-07-17T12:18:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.326100789Z", 2020-07-17T12:18:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_OI4YC_rV8-IvJU_ONTZMZ22", 2020-07-17T12:18:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.345795_174.1", 2020-07-17T12:18:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.345795397Z", 2020-07-17T12:18:10.610 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.611 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.612 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.351197_175.1", 2020-07-17T12:18:10.613 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.351197071Z", 2020-07-17T12:18:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.614 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.354716_176.1", 2020-07-17T12:18:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:10.617 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.354716106Z", 2020-07-17T12:18:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.618 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.619 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.360435_177.1", 2020-07-17T12:18:10.620 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.360435248Z", 2020-07-17T12:18:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.621 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:10.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.365795_178.1", 2020-07-17T12:18:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.365795374Z", 2020-07-17T12:18:10.624 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.625 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.626 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.372555_179.1", 2020-07-17T12:18:10.627 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.372555105Z", 2020-07-17T12:18:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.628 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.629 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.385196_180.1", 2020-07-17T12:18:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.631 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.385196882Z", 2020-07-17T12:18:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.632 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.633 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.393417_181.1", 2020-07-17T12:18:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.393417880Z", 2020-07-17T12:18:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.635 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.636 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.405697_182.1", 2020-07-17T12:18:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.405697153Z", 2020-07-17T12:18:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.640 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.415076_183.1", 2020-07-17T12:18:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.415076707Z", 2020-07-17T12:18:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.642 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.426554_184.1", 2020-07-17T12:18:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.645 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.426554025Z", 2020-07-17T12:18:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.646 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.647 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.648 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.433699_185.1", 2020-07-17T12:18:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.433699189Z", 2020-07-17T12:18:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.649 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.650 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.192609_186.1", 2020-07-17T12:18:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.652 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.192609023Z", 2020-07-17T12:18:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.653 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:10.654 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.197302_187.1", 2020-07-17T12:18:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.656 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.197302293Z", 2020-07-17T12:18:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.657 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.200146_188.1", 2020-07-17T12:18:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27", 2020-07-17T12:18:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.200146111Z", 2020-07-17T12:18:10.660 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.661 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_EAM3oic4BNiOLUongn3X0x8", 2020-07-17T12:18:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.662 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.204265_189.1", 2020-07-17T12:18:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.663 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27", 2020-07-17T12:18:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.204265093Z", 2020-07-17T12:18:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.664 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_EAM3oic4BNiOLUongn3X0x8", 2020-07-17T12:18:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.232604_190.1", 2020-07-17T12:18:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.232604283Z", 2020-07-17T12:18:10.667 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.668 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.236653_191.1", 2020-07-17T12:18:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.670 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.236653061Z", 2020-07-17T12:18:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.238308_192.1", 2020-07-17T12:18:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28", 2020-07-17T12:18:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.238308219Z", 2020-07-17T12:18:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_8tICQSSPhT4sUiRts2CGtXF", 2020-07-17T12:18:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.242121_193.1", 2020-07-17T12:18:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28", 2020-07-17T12:18:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.242121826Z", 2020-07-17T12:18:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_8tICQSSPhT4sUiRts2CGtXF", 2020-07-17T12:18:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.291427_194.1", 2020-07-17T12:18:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.291427038Z", 2020-07-17T12:18:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.296400_195.1", 2020-07-17T12:18:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.296400519Z", 2020-07-17T12:18:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.299100_196.1", 2020-07-17T12:18:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29", 2020-07-17T12:18:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.299100799Z", 2020-07-17T12:18:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oUHOV9TdQ-W-rj70_Cou9X5", 2020-07-17T12:18:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.304751_197.1", 2020-07-17T12:18:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29", 2020-07-17T12:18:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.304751236Z", 2020-07-17T12:18:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oUHOV9TdQ-W-rj70_Cou9X5", 2020-07-17T12:18:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.329636_198.1", 2020-07-17T12:18:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.329636096Z", 2020-07-17T12:18:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.334331_199.1", 2020-07-17T12:18:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.334331520Z", 2020-07-17T12:18:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.388479_200.1", 2020-07-17T12:18:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.388479756Z", 2020-07-17T12:18:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.398272_201.1", 2020-07-17T12:18:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.398272953Z", 2020-07-17T12:18:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.463878_202.1", 2020-07-17T12:18:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.463878801Z", 2020-07-17T12:18:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.474476_203.1", 2020-07-17T12:18:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.474476699Z", 2020-07-17T12:18:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.543823_204.1", 2020-07-17T12:18:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.543823468Z", 2020-07-17T12:18:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.548616_205.1", 2020-07-17T12:18:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.548616662Z", 2020-07-17T12:18:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.550563_206.1", 2020-07-17T12:18:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30", 2020-07-17T12:18:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.550563904Z", 2020-07-17T12:18:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_lJ2S3jRGkcD2jZEcb0bU-NY", 2020-07-17T12:18:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.555228_207.1", 2020-07-17T12:18:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30", 2020-07-17T12:18:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.555228975Z", 2020-07-17T12:18:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_lJ2S3jRGkcD2jZEcb0bU-NY", 2020-07-17T12:18:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.579434_208.1", 2020-07-17T12:18:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.579434456Z", 2020-07-17T12:18:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.583820_209.1", 2020-07-17T12:18:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.583820075Z", 2020-07-17T12:18:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.585899_210.1", 2020-07-17T12:18:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31", 2020-07-17T12:18:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.585899826Z", 2020-07-17T12:18:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_ln1eOUq8lDkVI4IpQvkNAi4", 2020-07-17T12:18:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.589417_211.1", 2020-07-17T12:18:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31", 2020-07-17T12:18:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.589417398Z", 2020-07-17T12:18:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_ln1eOUq8lDkVI4IpQvkNAi4", 2020-07-17T12:18:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.638248_212.1", 2020-07-17T12:18:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.638248979Z", 2020-07-17T12:18:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.642956_213.1", 2020-07-17T12:18:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.642956157Z", 2020-07-17T12:18:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.644834_214.1", 2020-07-17T12:18:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32", 2020-07-17T12:18:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.644834814Z", 2020-07-17T12:18:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_CTGkGtU8NTFdGu11wP4HBlE", 2020-07-17T12:18:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.649548_215.1", 2020-07-17T12:18:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32", 2020-07-17T12:18:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.649548275Z", 2020-07-17T12:18:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_CTGkGtU8NTFdGu11wP4HBlE", 2020-07-17T12:18:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.691484_216.1", 2020-07-17T12:18:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.691484060Z", 2020-07-17T12:18:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.695812_217.1", 2020-07-17T12:18:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.695812573Z", 2020-07-17T12:18:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.752305_218.1", 2020-07-17T12:18:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.752305251Z", 2020-07-17T12:18:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.759930_219.1", 2020-07-17T12:18:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.759930328Z", 2020-07-17T12:18:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.819822_220.1", 2020-07-17T12:18:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.819822109Z", 2020-07-17T12:18:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.829887_221.1", 2020-07-17T12:18:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.829887197Z", 2020-07-17T12:18:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.897711_222.1", 2020-07-17T12:18:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.897711412Z", 2020-07-17T12:18:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.901139_223.1", 2020-07-17T12:18:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.901139991Z", 2020-07-17T12:18:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.902679_224.1", 2020-07-17T12:18:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33", 2020-07-17T12:18:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.902679722Z", 2020-07-17T12:18:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_UxWO-19hsf0zfhgE2d7C0Bs", 2020-07-17T12:18:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.906270_225.1", 2020-07-17T12:18:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33", 2020-07-17T12:18:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.906270781Z", 2020-07-17T12:18:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_UxWO-19hsf0zfhgE2d7C0Bs", 2020-07-17T12:18:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.932251_226.1", 2020-07-17T12:18:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.794 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:10.794 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.932251093Z", 2020-07-17T12:18:10.795 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.795 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.795 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.937747_227.1", 2020-07-17T12:18:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.937747244Z", 2020-07-17T12:18:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.801 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.801 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.802 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:10.802 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.939489_228.1", 2020-07-17T12:18:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.805 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34", 2020-07-17T12:18:10.805 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.939489887Z", 2020-07-17T12:18:10.805 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.806 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.806 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.806 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oKhT8xShmiPT7_3DwLLaiZf", 2020-07-17T12:18:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.942587_229.1", 2020-07-17T12:18:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34", 2020-07-17T12:18:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.942587543Z", 2020-07-17T12:18:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.813 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.813 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.814 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oKhT8xShmiPT7_3DwLLaiZf", 2020-07-17T12:18:10.814 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.815 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.815 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.977552_230.1", 2020-07-17T12:18:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.977552760Z", 2020-07-17T12:18:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.818 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.818 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.818 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.819 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.819 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.982605_231.1", 2020-07-17T12:18:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.982605704Z", 2020-07-17T12:18:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.823 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.823 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.823 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.823 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.824 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:10.824 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.824 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.824 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.825 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.985248_232.1", 2020-07-17T12:18:10.826 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.826 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35", 2020-07-17T12:18:10.826 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.985248699Z", 2020-07-17T12:18:10.826 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.826 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.826 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.827 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.827 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.827 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_vxH0cpq2NUCS2-F5AwiLlGy", 2020-07-17T12:18:10.827 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.828 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.828 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.829 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.989554_233.1", 2020-07-17T12:18:10.829 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.829 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35", 2020-07-17T12:18:10.830 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.989554870Z", 2020-07-17T12:18:10.830 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.830 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.830 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.831 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.831 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.831 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_vxH0cpq2NUCS2-F5AwiLlGy", 2020-07-17T12:18:10.831 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.832 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.832 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.833 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988149.009800_234.1", 2020-07-17T12:18:10.833 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.833 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:10.833 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:49.009800265Z", 2020-07-17T12:18:10.834 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.834 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.834 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:10.834 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.835 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.835 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:10.835 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.836 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.836 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.837 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988149.014525_235.1", 2020-07-17T12:18:10.838 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.838 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:10.838 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:49.014525305Z", 2020-07-17T12:18:10.838 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.838 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.839 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:10.839 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.839 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.839 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:10.840 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.840 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.840 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.840 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.841 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988150.064957_236.1", 2020-07-17T12:18:10.841 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.841 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:10.842 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:50.064957041Z", 2020-07-17T12:18:10.842 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.842 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.842 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:10.843 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.843 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.844 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:10.844 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.844 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.845 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.846 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988150.071666_237.1", 2020-07-17T12:18:10.846 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.847 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:10.847 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:50.071666653Z", 2020-07-17T12:18:10.847 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.847 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.847 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:10.848 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.848 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.848 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:10.848 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.849 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.849 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.850 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.128899_238.1", 2020-07-17T12:18:10.850 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.850 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:10.850 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.128899145Z", 2020-07-17T12:18:10.850 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.851 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.851 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:10.851 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.852 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:10.852 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.853 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.853 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.854 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.136044_239.1", 2020-07-17T12:18:10.854 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.854 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:10.855 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.136044802Z", 2020-07-17T12:18:10.855 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.855 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.855 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:10.855 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.856 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:10.857 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:10.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.857 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.857 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.858 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.223773_240.1", 2020-07-17T12:18:10.858 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.859 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-36:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.36", 2020-07-17T12:18:10.859 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.223773094Z", 2020-07-17T12:18:10.859 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.859 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.859 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.860 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.860 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.860 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jdX6lkwwXVktMAEuRsLdL9f", 2020-07-17T12:18:10.860 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.861 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.861 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.862 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.227459_241.1", 2020-07-17T12:18:10.862 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.862 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-36:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.36", 2020-07-17T12:18:10.862 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.227459296Z", 2020-07-17T12:18:10.862 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.863 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.863 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.863 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.863 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.864 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jdX6lkwwXVktMAEuRsLdL9f", 2020-07-17T12:18:10.864 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.865 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.865 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.867 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.229174_242.1", 2020-07-17T12:18:10.867 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.867 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-36", 2020-07-17T12:18:10.868 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.229174371Z", 2020-07-17T12:18:10.868 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.868 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.868 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.868 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.869 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.869 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_qgrqKYBWaNJyRRn3Vjx0wVN", 2020-07-17T12:18:10.869 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.870 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.870 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.871 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.232729_243.1", 2020-07-17T12:18:10.871 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.871 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-36", 2020-07-17T12:18:10.872 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.232729435Z", 2020-07-17T12:18:10.872 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.872 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.872 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.873 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.873 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.873 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_qgrqKYBWaNJyRRn3Vjx0wVN", 2020-07-17T12:18:10.873 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.874 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.874 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.876 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.269955_244.1", 2020-07-17T12:18:10.876 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.876 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-37:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.37", 2020-07-17T12:18:10.876 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.269955795Z", 2020-07-17T12:18:10.877 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.877 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.877 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.878 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.878 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_c-tGp8YfI6B1yBpFUhIj44i", 2020-07-17T12:18:10.878 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.879 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.879 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.880 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.275803_245.1", 2020-07-17T12:18:10.880 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.880 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-37:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.37", 2020-07-17T12:18:10.880 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.275803241Z", 2020-07-17T12:18:10.880 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.881 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.881 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.881 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.881 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.882 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_c-tGp8YfI6B1yBpFUhIj44i", 2020-07-17T12:18:10.882 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.882 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.882 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.883 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.278648_246.1", 2020-07-17T12:18:10.884 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.884 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-37", 2020-07-17T12:18:10.884 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.278648356Z", 2020-07-17T12:18:10.884 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.884 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.885 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.885 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.885 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.885 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_9F_FGhwZ3EnXzZglCBoTwNA", 2020-07-17T12:18:10.886 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.886 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.886 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.887 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.887 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.285189_247.1", 2020-07-17T12:18:10.887 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.887 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-37", 2020-07-17T12:18:10.888 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.285189703Z", 2020-07-17T12:18:10.888 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.888 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.888 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.888 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.889 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_9F_FGhwZ3EnXzZglCBoTwNA", 2020-07-17T12:18:10.889 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.889 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.890 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.891 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.334180_248.1", 2020-07-17T12:18:10.891 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.891 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-38:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.38", 2020-07-17T12:18:10.891 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.334180801Z", 2020-07-17T12:18:10.891 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.892 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.892 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.892 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.892 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.893 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_hVnb1ngvk17nkYrOuLaCqwM", 2020-07-17T12:18:10.893 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.893 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.893 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.895 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.337583_249.1", 2020-07-17T12:18:10.895 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.895 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-38:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.38", 2020-07-17T12:18:10.895 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.337583158Z", 2020-07-17T12:18:10.895 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.895 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.896 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.896 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.897 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.897 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_hVnb1ngvk17nkYrOuLaCqwM", 2020-07-17T12:18:10.897 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.898 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.898 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.899 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.339086_250.1", 2020-07-17T12:18:10.899 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.900 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-38", 2020-07-17T12:18:10.900 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.339086250Z", 2020-07-17T12:18:10.900 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.900 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.901 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.901 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_3EGy4bfasMLEwsmYrKxh2v5", 2020-07-17T12:18:10.902 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.902 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.902 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.904 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.342294_251.1", 2020-07-17T12:18:10.904 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.904 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-38", 2020-07-17T12:18:10.904 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.342294371Z", 2020-07-17T12:18:10.904 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.905 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.905 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.905 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.905 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.905 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_3EGy4bfasMLEwsmYrKxh2v5", 2020-07-17T12:18:10.906 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.906 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.906 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.907 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.949522_252.1", 2020-07-17T12:18:10.907 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.908 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-39:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.39", 2020-07-17T12:18:10.908 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.949522585Z", 2020-07-17T12:18:10.908 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.908 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.908 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.909 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.909 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.909 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gsVvIhPoNrpxq4YxIZghI20", 2020-07-17T12:18:10.909 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.910 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.910 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.911 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.956039_253.1", 2020-07-17T12:18:10.911 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.911 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-39:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.39", 2020-07-17T12:18:10.911 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.956039623Z", 2020-07-17T12:18:10.912 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.912 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.912 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.912 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.913 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.913 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gsVvIhPoNrpxq4YxIZghI20", 2020-07-17T12:18:10.913 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.913 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.914 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.914 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.914 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.915 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.961169_254.1", 2020-07-17T12:18:10.915 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.915 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-39", 2020-07-17T12:18:10.915 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.961169072Z", 2020-07-17T12:18:10.915 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.916 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.916 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.916 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.916 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.917 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jHFRYvjwFaeKft8jqkFw9f6", 2020-07-17T12:18:10.917 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.917 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.917 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.918 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.965532_255.1", 2020-07-17T12:18:10.918 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.919 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-39", 2020-07-17T12:18:10.919 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.965532718Z", 2020-07-17T12:18:10.919 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.919 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.919 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.920 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.920 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.920 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jHFRYvjwFaeKft8jqkFw9f6", 2020-07-17T12:18:10.920 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.921 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.921 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.922 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.996911_256.1", 2020-07-17T12:18:10.922 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.922 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-40:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.40", 2020-07-17T12:18:10.922 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.996911090Z", 2020-07-17T12:18:10.923 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.923 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.923 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.924 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.924 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zd9daTaLaaQPrwIpjVSY9v4", 2020-07-17T12:18:10.924 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.924 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.924 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.925 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.001607_257.1", 2020-07-17T12:18:10.926 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.926 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-40:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.40", 2020-07-17T12:18:10.926 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.001607160Z", 2020-07-17T12:18:10.926 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.926 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.927 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.927 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.927 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.927 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zd9daTaLaaQPrwIpjVSY9v4", 2020-07-17T12:18:10.927 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.928 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.928 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.929 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.004358_258.1", 2020-07-17T12:18:10.929 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.929 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-40", 2020-07-17T12:18:10.930 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.004358398Z", 2020-07-17T12:18:10.930 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.930 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.930 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.931 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Apkycxzmtorr0IIi3gzU-Qm", 2020-07-17T12:18:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.009492_259.1", 2020-07-17T12:18:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-40", 2020-07-17T12:18:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.009492644Z", 2020-07-17T12:18:10.934 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.935 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Apkycxzmtorr0IIi3gzU-Qm", 2020-07-17T12:18:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.048556_260.1", 2020-07-17T12:18:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.937 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-41:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.41", 2020-07-17T12:18:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.048556572Z", 2020-07-17T12:18:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.938 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NntBaMu1tW2iuZqebtOm9OP", 2020-07-17T12:18:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.939 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.053032_261.1", 2020-07-17T12:18:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-41:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.41", 2020-07-17T12:18:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.053032571Z", 2020-07-17T12:18:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.941 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.942 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NntBaMu1tW2iuZqebtOm9OP", 2020-07-17T12:18:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.055326_262.1", 2020-07-17T12:18:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.944 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-41", 2020-07-17T12:18:10.945 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.055326711Z", 2020-07-17T12:18:10.945 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.945 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.945 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.945 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_W8htjS8YlHsEASKVqW9GGew", 2020-07-17T12:18:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.060681_263.1", 2020-07-17T12:18:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-41", 2020-07-17T12:18:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.060681987Z", 2020-07-17T12:18:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.948 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.949 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_W8htjS8YlHsEASKVqW9GGew", 2020-07-17T12:18:10.950 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.950 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.950 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.299855_42.1", 2020-07-17T12:18:10.951 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-1:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.1", 2020-07-17T12:18:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.299855768Z", 2020-07-17T12:18:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.952 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fCYmrES933MNW-ZftW97tAH", 2020-07-17T12:18:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.318362_43.1", 2020-07-17T12:18:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-1:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.1", 2020-07-17T12:18:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.318362600Z", 2020-07-17T12:18:10.955 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.956 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.956 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.956 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.956 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.957 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fCYmrES933MNW-ZftW97tAH", 2020-07-17T12:18:10.957 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.957 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.958 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.958 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.958 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.958 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.322473_44.1", 2020-07-17T12:18:10.959 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.959 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-1", 2020-07-17T12:18:10.959 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.322473615Z", 2020-07-17T12:18:10.959 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.960 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.960 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.960 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.960 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.960 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_N3sIyaABoPKhvVJDTubfANC", 2020-07-17T12:18:10.961 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.961 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.961 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.962 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.326365_45.1", 2020-07-17T12:18:10.962 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.963 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-1", 2020-07-17T12:18:10.963 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.326365120Z", 2020-07-17T12:18:10.963 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.963 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.963 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.963 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.964 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.964 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_N3sIyaABoPKhvVJDTubfANC", 2020-07-17T12:18:10.964 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.965 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.965 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.966 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.371051_46.1", 2020-07-17T12:18:10.967 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.967 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-2:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.2", 2020-07-17T12:18:10.967 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.371051231Z", 2020-07-17T12:18:10.967 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.967 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.968 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.968 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.968 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.968 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_bpM5KcuqzRHhKrpzuuAWOQK", 2020-07-17T12:18:10.969 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.969 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.969 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.970 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.376155_47.1", 2020-07-17T12:18:10.970 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.970 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-2:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.2", 2020-07-17T12:18:10.971 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.376155577Z", 2020-07-17T12:18:10.971 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.971 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.971 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.971 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.972 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.972 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_bpM5KcuqzRHhKrpzuuAWOQK", 2020-07-17T12:18:10.972 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.972 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.973 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.974 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.378460_48.1", 2020-07-17T12:18:10.974 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.974 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-2", 2020-07-17T12:18:10.974 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.378460095Z", 2020-07-17T12:18:10.974 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.975 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.975 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.975 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.975 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.975 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_o0AcCUXo7hxp6mur0vpSEA6", 2020-07-17T12:18:10.976 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.976 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.976 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.977 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.382370_49.1", 2020-07-17T12:18:10.977 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.978 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-2", 2020-07-17T12:18:10.978 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.382370235Z", 2020-07-17T12:18:10.978 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.978 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.978 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.978 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.979 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_o0AcCUXo7hxp6mur0vpSEA6", 2020-07-17T12:18:10.979 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.980 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.980 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.981 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987598.171972_50.1", 2020-07-17T12:18:10.981 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.981 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-3:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.3", 2020-07-17T12:18:10.981 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:38.171972137Z", 2020-07-17T12:18:10.981 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.982 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.982 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.982 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.982 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.983 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jXpmpU1bMKc6cP4eCcdC0E5", 2020-07-17T12:18:10.983 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.983 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.983 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.983 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.984 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.984 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987598.175455_51.1", 2020-07-17T12:18:10.984 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.985 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-3:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.3", 2020-07-17T12:18:10.985 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:38.175455638Z", 2020-07-17T12:18:10.985 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.985 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.985 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jXpmpU1bMKc6cP4eCcdC0E5", 2020-07-17T12:18:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.988 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987598.177003_52.1", 2020-07-17T12:18:10.988 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.988 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-3", 2020-07-17T12:18:10.988 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:38.177003803Z", 2020-07-17T12:18:10.988 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.989 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.989 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.989 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.989 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.989 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_-b65PCt1-zecno94kbqdZQV", 2020-07-17T12:18:10.990 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.990 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.990 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.991 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987598.180439_53.1", 2020-07-17T12:18:10.991 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:10.991 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-3", 2020-07-17T12:18:10.992 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:38.180439200Z", 2020-07-17T12:18:10.992 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.992 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.992 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.993 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.993 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.993 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_-b65PCt1-zecno94kbqdZQV", 2020-07-17T12:18:10.993 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.994 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.994 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:10.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.995 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.433011_54.1", 2020-07-17T12:18:10.995 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.995 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-4:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.4", 2020-07-17T12:18:10.996 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.433011959Z", 2020-07-17T12:18:10.996 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.996 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:10.996 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:10.996 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:10.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.997 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:10.997 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__ea-Segy13dxgNiJPXZ7Zd9", 2020-07-17T12:18:10.997 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:10.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.997 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:10.998 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:10.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:10.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:10.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:10.999 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.438333_55.1", 2020-07-17T12:18:10.999 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:10.999 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-4:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.4", 2020-07-17T12:18:10.999 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.438333452Z", 2020-07-17T12:18:10.999 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:10.999 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.000 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.000 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.000 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.000 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__ea-Segy13dxgNiJPXZ7Zd9", 2020-07-17T12:18:11.000 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.001 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.001 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.002 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.440677_56.1", 2020-07-17T12:18:11.002 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.002 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-4", 2020-07-17T12:18:11.002 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.440677898Z", 2020-07-17T12:18:11.003 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.003 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.003 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.003 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.004 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.004 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_WRyLUNBmhHEnabp7o-DhlbH", 2020-07-17T12:18:11.004 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.004 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.005 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.005 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.005 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.006 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.445531_57.1", 2020-07-17T12:18:11.006 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.006 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-4", 2020-07-17T12:18:11.006 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.445531158Z", 2020-07-17T12:18:11.006 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.006 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.007 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.007 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.007 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.007 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_WRyLUNBmhHEnabp7o-DhlbH", 2020-07-17T12:18:11.008 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.008 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.008 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.468350_58.1", 2020-07-17T12:18:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-5:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.5", 2020-07-17T12:18:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.468350431Z", 2020-07-17T12:18:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_FMcLIdx8RNwH2qOVrLRBKLI", 2020-07-17T12:18:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.472556_59.1", 2020-07-17T12:18:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-5:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.5", 2020-07-17T12:18:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.472556928Z", 2020-07-17T12:18:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_FMcLIdx8RNwH2qOVrLRBKLI", 2020-07-17T12:18:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.474186_60.1", 2020-07-17T12:18:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-5", 2020-07-17T12:18:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.474186224Z", 2020-07-17T12:18:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_7oOlSD0kLzDgbQh79FmiwVN", 2020-07-17T12:18:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.020 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.477662_61.1", 2020-07-17T12:18:11.020 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.020 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-5", 2020-07-17T12:18:11.020 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.477662122Z", 2020-07-17T12:18:11.021 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.021 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.021 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.021 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.022 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.022 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_7oOlSD0kLzDgbQh79FmiwVN", 2020-07-17T12:18:11.022 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.022 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.022 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.023 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.023 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.023 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.518094_62.1", 2020-07-17T12:18:11.024 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.024 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-6:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.6", 2020-07-17T12:18:11.024 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.518094230Z", 2020-07-17T12:18:11.024 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.024 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.025 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.025 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.025 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.025 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_98i-mz1lveZeu2xWlqfjP8v", 2020-07-17T12:18:11.026 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.026 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.026 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.027 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.525483_63.1", 2020-07-17T12:18:11.027 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.027 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-6:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.6", 2020-07-17T12:18:11.028 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.525483008Z", 2020-07-17T12:18:11.028 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.028 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.028 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.028 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.029 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.029 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_98i-mz1lveZeu2xWlqfjP8v", 2020-07-17T12:18:11.029 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.029 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.030 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.030 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.030 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.031 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.527486_64.1", 2020-07-17T12:18:11.031 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.031 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-6", 2020-07-17T12:18:11.031 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.527486008Z", 2020-07-17T12:18:11.031 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.032 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.032 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.032 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.032 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.033 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6KmjjH6ijuVTEeEq7jAXi7H", 2020-07-17T12:18:11.033 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.033 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.033 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.034 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.533990_65.1", 2020-07-17T12:18:11.035 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.035 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-6", 2020-07-17T12:18:11.035 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.533990618Z", 2020-07-17T12:18:11.035 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.035 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.035 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.036 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.036 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.036 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6KmjjH6ijuVTEeEq7jAXi7H", 2020-07-17T12:18:11.036 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.037 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.037 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.038 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.314142_66.1", 2020-07-17T12:18:11.038 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.038 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.7", 2020-07-17T12:18:11.038 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.314142380Z", 2020-07-17T12:18:11.039 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.039 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.039 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.039 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.039 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.040 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_3p7wlqKX2jIqNSa48PpGSPB", 2020-07-17T12:18:11.040 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.040 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.040 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.041 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.321643_67.1", 2020-07-17T12:18:11.041 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.042 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.7", 2020-07-17T12:18:11.042 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.321643612Z", 2020-07-17T12:18:11.042 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.042 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.042 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.043 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.043 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.043 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_3p7wlqKX2jIqNSa48PpGSPB", 2020-07-17T12:18:11.043 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.044 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.044 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.045 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.324252_68.1", 2020-07-17T12:18:11.045 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.045 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7", 2020-07-17T12:18:11.045 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.324252955Z", 2020-07-17T12:18:11.045 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.046 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.046 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.046 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.046 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.047 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:11.047 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.047 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.047 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.048 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.329888_69.1", 2020-07-17T12:18:11.048 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.049 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7", 2020-07-17T12:18:11.049 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.329888896Z", 2020-07-17T12:18:11.049 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.049 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.049 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.050 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.050 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.050 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:11.050 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.051 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.051 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.052 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.360390_70.1", 2020-07-17T12:18:11.052 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.052 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.8", 2020-07-17T12:18:11.052 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.360390239Z", 2020-07-17T12:18:11.052 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.053 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.053 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.053 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.053 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.054 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r09s2aK_hqiAA214GJtpxF1", 2020-07-17T12:18:11.054 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.054 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.054 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.055 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.364581_71.1", 2020-07-17T12:18:11.055 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.056 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.8", 2020-07-17T12:18:11.056 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.364581063Z", 2020-07-17T12:18:11.056 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.056 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.056 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.057 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.057 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.057 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r09s2aK_hqiAA214GJtpxF1", 2020-07-17T12:18:11.057 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.058 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.058 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.059 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.369486_72.1", 2020-07-17T12:18:11.059 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.059 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8", 2020-07-17T12:18:11.060 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.369486518Z", 2020-07-17T12:18:11.060 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.060 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.060 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.060 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.061 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.061 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:11.061 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.062 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.062 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.062 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.062 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.063 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.373949_73.1", 2020-07-17T12:18:11.063 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.063 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8", 2020-07-17T12:18:11.063 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.373949060Z", 2020-07-17T12:18:11.063 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.064 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.064 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.064 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.064 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.065 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:11.065 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.065 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.065 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.066 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.424920_75.1", 2020-07-17T12:18:11.067 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.067 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.9", 2020-07-17T12:18:11.067 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.424920583Z", 2020-07-17T12:18:11.067 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.067 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.067 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.068 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.068 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_znrNp-QgDthcyR2ClROORQe", 2020-07-17T12:18:11.068 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.069 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.069 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.070 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.431645_76.1", 2020-07-17T12:18:11.070 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.070 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.9", 2020-07-17T12:18:11.070 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.431645809Z", 2020-07-17T12:18:11.071 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.071 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.071 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.071 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.071 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.072 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_znrNp-QgDthcyR2ClROORQe", 2020-07-17T12:18:11.072 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.072 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.072 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.074 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.434319_77.1", 2020-07-17T12:18:11.074 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.074 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9", 2020-07-17T12:18:11.074 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.434319003Z", 2020-07-17T12:18:11.074 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.075 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.075 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.075 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.075 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.076 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:11.076 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.076 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.076 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:11.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.077 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.439076_78.1", 2020-07-17T12:18:11.077 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.078 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9", 2020-07-17T12:18:11.078 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.439076836Z", 2020-07-17T12:18:11.078 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.078 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.078 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:11.078 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:11.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.079 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.079 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:11.079 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.080 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.080 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.081 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.701989_80.1", 2020-07-17T12:18:11.081 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.081 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7", 2020-07-17T12:18:11.081 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.701989372Z", 2020-07-17T12:18:11.081 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.082 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.082 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:11.082 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.082 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.083 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:11.083 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:11.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.083 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.083 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "remove" 2020-07-17T12:18:11.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.084 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.708007_81.1", 2020-07-17T12:18:11.084 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.085 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7", 2020-07-17T12:18:11.085 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.708007772Z", 2020-07-17T12:18:11.085 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.085 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.085 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:11.086 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.086 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.086 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:11.086 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:11.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.087 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.087 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.088 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.781097_82.1", 2020-07-17T12:18:11.088 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.088 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8", 2020-07-17T12:18:11.088 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.781097190Z", 2020-07-17T12:18:11.088 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.089 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:11.089 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.089 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:11.090 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:11.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.090 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.090 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "remove" 2020-07-17T12:18:11.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.091 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.786755_83.1", 2020-07-17T12:18:11.091 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.092 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8", 2020-07-17T12:18:11.092 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.786755007Z", 2020-07-17T12:18:11.092 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.092 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.092 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:11.092 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.093 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.094 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:11.094 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:11.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.095 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.095 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.096 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.889801_85.1", 2020-07-17T12:18:11.096 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.096 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9", 2020-07-17T12:18:11.096 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.889801769Z", 2020-07-17T12:18:11.096 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.097 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.097 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:11.097 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.097 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.098 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:11.098 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:11.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.098 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.098 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "remove" 2020-07-17T12:18:11.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:11.099 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.895710_86.1", 2020-07-17T12:18:11.099 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:11.100 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9", 2020-07-17T12:18:11.100 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710719Z", 2020-07-17T12:18:11.100 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:11.100 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:11.100 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:11.100 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:11.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.101 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:11.101 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:11.101 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:11.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:11.102 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:11.102 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:11.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:11.102 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:11.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:11.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:11.106 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 34954458-202e-48c3-864b-c00b15f41dd5 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:11.143 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:11.144 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:11.353 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:18:11.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.354 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987471.882144_44.1", 2020-07-17T12:18:11.354 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "user", 2020-07-17T12:18:11.354 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw-multisite-test-user", 2020-07-17T12:18:11.355 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.882144008Z", 2020-07-17T12:18:11.355 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.355 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.355 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.355 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.356 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.356 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_uJ2kpFkaIMR3kSeSIGfEB3T", 2020-07-17T12:18:11.356 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.357 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.357 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987471.890977_45.1", 2020-07-17T12:18:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "user", 2020-07-17T12:18:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw-multisite-test-user", 2020-07-17T12:18:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977275Z", 2020-07-17T12:18:11.359 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.359 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.359 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.359 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.360 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.360 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_uJ2kpFkaIMR3kSeSIGfEB3T", 2020-07-17T12:18:11.360 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.360 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.360 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.361 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.172466_48.1", 2020-07-17T12:18:11.362 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.362 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-10:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.10", 2020-07-17T12:18:11.362 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.172466793Z", 2020-07-17T12:18:11.362 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.362 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.363 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.363 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.363 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.364 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_mG7u6kOIVSSgWYLweK5uzvb", 2020-07-17T12:18:11.364 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.364 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.364 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.365 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.365 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.177556_49.1", 2020-07-17T12:18:11.365 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.366 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-10:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.10", 2020-07-17T12:18:11.366 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.177556387Z", 2020-07-17T12:18:11.366 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.366 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.366 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.367 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.367 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.367 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_mG7u6kOIVSSgWYLweK5uzvb", 2020-07-17T12:18:11.367 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.368 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.368 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.369 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.182179_50.1", 2020-07-17T12:18:11.369 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.369 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-10", 2020-07-17T12:18:11.370 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.182179376Z", 2020-07-17T12:18:11.370 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.370 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.370 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.370 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.371 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.371 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_yuMwADwMvlFMi5Y6H00iObr", 2020-07-17T12:18:11.371 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.371 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.372 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.372 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.372 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.372 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.186476_51.1", 2020-07-17T12:18:11.373 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.373 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-10", 2020-07-17T12:18:11.373 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.186476843Z", 2020-07-17T12:18:11.373 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.373 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.374 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.374 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.374 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.374 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_yuMwADwMvlFMi5Y6H00iObr", 2020-07-17T12:18:11.374 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.375 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.375 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.375 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.375 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.376 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.376 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.216713_52.1", 2020-07-17T12:18:11.376 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.376 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-11:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.11", 2020-07-17T12:18:11.376 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.216713690Z", 2020-07-17T12:18:11.377 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.377 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.377 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.377 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.378 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.378 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gq5MHLpwjP0npYemQW4217g", 2020-07-17T12:18:11.378 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.378 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.378 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.379 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.379 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.379 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.222092_53.1", 2020-07-17T12:18:11.380 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.380 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-11:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.11", 2020-07-17T12:18:11.380 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.222092496Z", 2020-07-17T12:18:11.380 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.380 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.380 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.381 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.381 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.381 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.381 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gq5MHLpwjP0npYemQW4217g", 2020-07-17T12:18:11.381 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.382 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.385 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.385 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.386 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.386 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.386 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.223732_54.1", 2020-07-17T12:18:11.386 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.387 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-11", 2020-07-17T12:18:11.387 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.223732073Z", 2020-07-17T12:18:11.387 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.387 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.387 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.388 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.388 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.388 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_hmDQZxZXEQLZ3GX5-Ia1DXm", 2020-07-17T12:18:11.388 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.389 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.389 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.390 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.390 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.227773_55.1", 2020-07-17T12:18:11.390 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.390 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-11", 2020-07-17T12:18:11.398 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.227773848Z", 2020-07-17T12:18:11.399 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.399 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.399 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.399 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.400 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.400 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_hmDQZxZXEQLZ3GX5-Ia1DXm", 2020-07-17T12:18:11.400 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.401 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.401 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.401 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.401 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.402 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.263759_56.1", 2020-07-17T12:18:11.402 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.402 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-12:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.12", 2020-07-17T12:18:11.402 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.263759779Z", 2020-07-17T12:18:11.402 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.403 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.403 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.403 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.403 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.404 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_kJPn3awfkoAAv2YoILjX1YX", 2020-07-17T12:18:11.404 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.404 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.404 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.405 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.405 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.272286_57.1", 2020-07-17T12:18:11.405 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.406 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-12:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.12", 2020-07-17T12:18:11.406 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.272286467Z", 2020-07-17T12:18:11.406 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.406 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.406 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.406 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.407 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.407 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_kJPn3awfkoAAv2YoILjX1YX", 2020-07-17T12:18:11.407 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.408 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.408 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.408 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.409 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.274467_58.1", 2020-07-17T12:18:11.409 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.409 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-12", 2020-07-17T12:18:11.409 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.274467332Z", 2020-07-17T12:18:11.409 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.410 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.410 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.410 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.410 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.410 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_WFmqiEkT2ufoxFeioom356B", 2020-07-17T12:18:11.411 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.411 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.411 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.411 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.411 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.412 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.278144_59.1", 2020-07-17T12:18:11.412 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.412 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-12", 2020-07-17T12:18:11.413 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.278144599Z", 2020-07-17T12:18:11.413 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.413 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.413 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.413 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.414 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.414 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_WFmqiEkT2ufoxFeioom356B", 2020-07-17T12:18:11.414 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.414 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.415 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.415 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.712640_60.1", 2020-07-17T12:18:11.416 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.416 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-13:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.13", 2020-07-17T12:18:11.416 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.712640382Z", 2020-07-17T12:18:11.416 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.416 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.417 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.417 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.417 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.417 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_coVsWkNqvKBUCWs8AacIoNl", 2020-07-17T12:18:11.417 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.418 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.418 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.419 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.717263_61.1", 2020-07-17T12:18:11.419 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.419 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-13:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.13", 2020-07-17T12:18:11.419 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.717263960Z", 2020-07-17T12:18:11.420 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.420 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.420 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.420 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.421 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.421 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_coVsWkNqvKBUCWs8AacIoNl", 2020-07-17T12:18:11.421 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.421 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.421 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.422 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.719823_62.1", 2020-07-17T12:18:11.422 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.423 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-13", 2020-07-17T12:18:11.423 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.719823082Z", 2020-07-17T12:18:11.423 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.423 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.423 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.424 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.424 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.424 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dWoVeSt6MXyutBS6iBOB_SN", 2020-07-17T12:18:11.424 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.425 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.425 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.425 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.425 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.426 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.725332_63.1", 2020-07-17T12:18:11.426 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.426 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-13", 2020-07-17T12:18:11.426 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.725332531Z", 2020-07-17T12:18:11.426 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.427 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.427 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.427 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.427 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.427 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dWoVeSt6MXyutBS6iBOB_SN", 2020-07-17T12:18:11.428 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.428 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.428 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.428 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.429 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.429 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.756675_64.1", 2020-07-17T12:18:11.429 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.429 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-14:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.14", 2020-07-17T12:18:11.430 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.756675726Z", 2020-07-17T12:18:11.430 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.430 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.430 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.430 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.431 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.431 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_BWPfn12CXl2TlU4AWhdAySv", 2020-07-17T12:18:11.431 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.431 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.432 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.432 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.432 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.433 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.761544_65.1", 2020-07-17T12:18:11.433 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.433 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-14:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.14", 2020-07-17T12:18:11.433 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.761544051Z", 2020-07-17T12:18:11.433 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.433 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.434 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.434 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.434 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.434 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_BWPfn12CXl2TlU4AWhdAySv", 2020-07-17T12:18:11.435 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.435 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.435 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.436 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.436 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.763526_66.1", 2020-07-17T12:18:11.436 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.437 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-14", 2020-07-17T12:18:11.437 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.763526945Z", 2020-07-17T12:18:11.437 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.437 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.437 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.438 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.438 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.438 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_irk8P5j0CkMCoEl6rP2eANQ", 2020-07-17T12:18:11.438 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.439 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.439 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.439 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.439 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.440 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.440 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.767394_67.1", 2020-07-17T12:18:11.440 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.440 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-14", 2020-07-17T12:18:11.440 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.767394556Z", 2020-07-17T12:18:11.441 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.441 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.441 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.441 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.442 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.442 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_irk8P5j0CkMCoEl6rP2eANQ", 2020-07-17T12:18:11.442 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.442 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.442 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.443 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.443 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.810791_68.1", 2020-07-17T12:18:11.444 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.444 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-15:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.15", 2020-07-17T12:18:11.444 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.810791998Z", 2020-07-17T12:18:11.444 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.444 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.444 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.445 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.445 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.445 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_aYP5BWVqE2ZQzg8KQUMoKIk", 2020-07-17T12:18:11.445 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.446 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.446 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.446 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.446 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.447 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.816286_69.1", 2020-07-17T12:18:11.447 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.447 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-15:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.15", 2020-07-17T12:18:11.447 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.816286566Z", 2020-07-17T12:18:11.448 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.448 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.448 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.448 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.449 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.449 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_aYP5BWVqE2ZQzg8KQUMoKIk", 2020-07-17T12:18:11.449 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.449 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.449 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.450 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.450 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.450 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.450 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.820905_70.1", 2020-07-17T12:18:11.451 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.451 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-15", 2020-07-17T12:18:11.451 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.820905407Z", 2020-07-17T12:18:11.451 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.451 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.451 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.452 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.452 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.452 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_rpYWMBKFNIdTSlnC72SKitx", 2020-07-17T12:18:11.452 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.453 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.453 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.453 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.453 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.454 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.830894_71.1", 2020-07-17T12:18:11.454 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.454 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-15", 2020-07-17T12:18:11.454 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.830894206Z", 2020-07-17T12:18:11.455 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.455 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.455 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.455 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.455 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.456 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_rpYWMBKFNIdTSlnC72SKitx", 2020-07-17T12:18:11.456 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.456 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.456 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.457 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.457 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.457 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.232376_72.1", 2020-07-17T12:18:11.457 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.458 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:11.458 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.232376434Z", 2020-07-17T12:18:11.458 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.458 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.458 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.459 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.459 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.459 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:11.459 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.460 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.460 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.461 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.236895_73.1", 2020-07-17T12:18:11.461 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.461 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:11.461 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.236895263Z", 2020-07-17T12:18:11.461 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.462 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.462 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.462 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.462 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.463 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:11.463 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.463 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.463 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.464 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.238681_74.1", 2020-07-17T12:18:11.465 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.465 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16", 2020-07-17T12:18:11.465 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.238681635Z", 2020-07-17T12:18:11.465 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.465 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.466 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.466 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.466 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.466 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_lEb4NYqynMKhYQrCh3ur9Pg", 2020-07-17T12:18:11.466 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.467 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.467 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.467 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.468 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.468 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.242366_75.1", 2020-07-17T12:18:11.468 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.469 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16", 2020-07-17T12:18:11.469 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.242366508Z", 2020-07-17T12:18:11.469 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.469 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.469 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.469 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.470 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.470 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_lEb4NYqynMKhYQrCh3ur9Pg", 2020-07-17T12:18:11.470 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.471 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.471 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.471 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.471 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.472 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.269805_76.1", 2020-07-17T12:18:11.472 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.472 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:11.472 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.269805039Z", 2020-07-17T12:18:11.472 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.473 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.473 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.473 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.473 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.473 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:11.474 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.474 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.474 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.474 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.475 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.475 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.475 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.273253_77.1", 2020-07-17T12:18:11.475 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.475 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:11.476 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.273253052Z", 2020-07-17T12:18:11.476 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.476 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.476 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.476 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.477 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.477 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:11.477 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.478 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.478 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.478 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.478 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.479 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.274824_78.1", 2020-07-17T12:18:11.479 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.479 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17", 2020-07-17T12:18:11.479 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.274824384Z", 2020-07-17T12:18:11.479 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.479 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.480 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.480 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.480 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.480 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_kC34LrBQXhenfmJdKuMdgnY", 2020-07-17T12:18:11.481 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.481 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.481 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.482 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.277982_79.1", 2020-07-17T12:18:11.482 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.482 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17", 2020-07-17T12:18:11.483 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.277982524Z", 2020-07-17T12:18:11.483 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.483 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.483 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.483 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.484 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.484 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_kC34LrBQXhenfmJdKuMdgnY", 2020-07-17T12:18:11.484 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.484 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.485 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.485 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.485 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.321344_80.1", 2020-07-17T12:18:11.486 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.486 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:11.486 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.321344326Z", 2020-07-17T12:18:11.486 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.486 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.487 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.487 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.487 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.487 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:11.487 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.488 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.488 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.489 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.325975_81.1", 2020-07-17T12:18:11.489 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.489 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:11.489 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.325975985Z", 2020-07-17T12:18:11.490 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.490 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.490 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.490 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.490 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.491 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:11.491 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.491 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.491 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.492 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.327813_82.1", 2020-07-17T12:18:11.492 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.493 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18", 2020-07-17T12:18:11.493 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.327813080Z", 2020-07-17T12:18:11.493 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.493 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.493 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.493 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.494 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.494 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_0c-EMQmUZNyncsOLFYYvBXZ", 2020-07-17T12:18:11.494 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.495 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.495 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.496 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.331302_83.1", 2020-07-17T12:18:11.496 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.496 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18", 2020-07-17T12:18:11.496 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.331302239Z", 2020-07-17T12:18:11.496 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.497 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.497 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.497 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.497 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.497 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_0c-EMQmUZNyncsOLFYYvBXZ", 2020-07-17T12:18:11.498 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.498 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.498 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.498 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.498 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.499 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.349661_84.1", 2020-07-17T12:18:11.499 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.499 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:11.500 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.349661555Z", 2020-07-17T12:18:11.500 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.500 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.500 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:11.500 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.501 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.501 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:11.501 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.501 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.502 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.502 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.502 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.502 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.354642_85.1", 2020-07-17T12:18:11.503 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.503 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:11.503 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.354642695Z", 2020-07-17T12:18:11.503 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.503 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.504 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:11.504 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.504 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.504 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:11.505 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.505 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.505 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.506 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.358809_86.1", 2020-07-17T12:18:11.506 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.506 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:11.507 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.358809552Z", 2020-07-17T12:18:11.507 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.507 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.507 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:11.507 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.508 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.508 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:11.508 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.508 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.509 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.509 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.364350_87.1", 2020-07-17T12:18:11.510 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.510 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:11.510 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.364350896Z", 2020-07-17T12:18:11.510 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.510 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.511 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:11.511 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.511 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.511 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:11.511 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.512 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.512 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.512 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.512 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.513 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.374842_88.1", 2020-07-17T12:18:11.513 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.513 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:11.513 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.374842199Z", 2020-07-17T12:18:11.514 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.514 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.514 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:11.514 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.514 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.515 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:11.515 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.515 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.515 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.516 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.516 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.516 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.381436_89.1", 2020-07-17T12:18:11.516 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.517 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:11.517 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.381436296Z", 2020-07-17T12:18:11.517 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.517 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.517 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:11.518 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.518 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.518 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:11.518 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.519 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.519 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.520 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.580189_90.1", 2020-07-17T12:18:11.520 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.520 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:11.520 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.580189016Z", 2020-07-17T12:18:11.520 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.521 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.521 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.521 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.521 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.521 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:11.522 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.522 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.522 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.522 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.522 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.523 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.523 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.583928_91.1", 2020-07-17T12:18:11.523 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.523 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:11.524 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.583928438Z", 2020-07-17T12:18:11.524 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.524 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.524 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.524 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.525 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.525 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:11.525 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.525 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.526 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.526 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.585595_92.1", 2020-07-17T12:18:11.527 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.527 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19", 2020-07-17T12:18:11.527 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.585595449Z", 2020-07-17T12:18:11.527 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.527 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.528 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.528 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.528 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.528 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Y0anvog_R4e6DQW9utotNY_", 2020-07-17T12:18:11.528 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.529 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.529 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.529 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.529 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.530 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.589056_93.1", 2020-07-17T12:18:11.530 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.530 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19", 2020-07-17T12:18:11.530 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.589056210Z", 2020-07-17T12:18:11.531 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.531 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.531 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.531 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.531 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.532 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Y0anvog_R4e6DQW9utotNY_", 2020-07-17T12:18:11.532 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.532 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.532 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.533 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.596888_94.1", 2020-07-17T12:18:11.533 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.534 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:11.534 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.596888308Z", 2020-07-17T12:18:11.534 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.534 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.534 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:11.534 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.535 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.535 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:11.535 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.536 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.536 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.536 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.536 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.537 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.602473_95.1", 2020-07-17T12:18:11.537 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.537 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:11.537 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.602473621Z", 2020-07-17T12:18:11.538 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.538 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.538 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:11.538 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.539 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.539 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:11.539 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.539 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.539 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.540 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987998.537142_96.1", 2020-07-17T12:18:11.541 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.541 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:11.541 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.537142107Z", 2020-07-17T12:18:11.541 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.541 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.541 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.542 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.542 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.542 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.542 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.543 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.543 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.543 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.543 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.544 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987998.541926_97.1", 2020-07-17T12:18:11.544 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.544 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:11.544 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.541926282Z", 2020-07-17T12:18:11.544 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.545 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.545 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.545 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.545 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.546 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.546 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.546 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.546 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.547 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.547 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987998.544300_98.1", 2020-07-17T12:18:11.547 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.548 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20", 2020-07-17T12:18:11.548 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.544300322Z", 2020-07-17T12:18:11.548 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.548 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.548 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.548 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.549 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.549 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_qoKh-ps-xZklh5sIuAPFkjZ", 2020-07-17T12:18:11.549 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.549 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.550 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.551 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987998.548993_99.1", 2020-07-17T12:18:11.551 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.551 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20", 2020-07-17T12:18:11.551 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.548993097Z", 2020-07-17T12:18:11.551 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.552 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.552 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.552 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.552 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.552 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_qoKh-ps-xZklh5sIuAPFkjZ", 2020-07-17T12:18:11.553 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.553 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.553 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.554 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.554 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.554 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.554 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988004.926158_100.1", 2020-07-17T12:18:11.554 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.555 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:11.555 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:24.926158716Z", 2020-07-17T12:18:11.555 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.555 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.555 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.556 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.556 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.556 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.556 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.557 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.557 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.557 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.557 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.558 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988004.936521_101.1", 2020-07-17T12:18:11.558 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.558 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:11.559 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:24.936521959Z", 2020-07-17T12:18:11.559 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.559 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.559 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.559 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.560 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.560 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.560 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.560 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.561 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.562 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988065.046888_102.1", 2020-07-17T12:18:11.562 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.562 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:11.562 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:25.046888654Z", 2020-07-17T12:18:11.562 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.563 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.563 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.563 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.563 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.564 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.564 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:11.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.564 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.564 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.565 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.565 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.566 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988065.056306_103.1", 2020-07-17T12:18:11.566 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.566 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:11.566 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:25.056306412Z", 2020-07-17T12:18:11.566 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.567 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.567 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.567 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.567 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.568 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:11.568 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:11.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.568 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.568 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.570 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.108561_114.1", 2020-07-17T12:18:11.570 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.570 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:11.570 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.108561553Z", 2020-07-17T12:18:11.570 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.571 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.571 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.571 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.571 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.572 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:11.572 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.572 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.572 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.573 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.573 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.573 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.112561_115.1", 2020-07-17T12:18:11.574 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.574 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:11.574 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.112561600Z", 2020-07-17T12:18:11.574 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.574 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.574 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.575 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.575 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.575 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:11.575 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.576 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.576 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.577 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.114341_116.1", 2020-07-17T12:18:11.577 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.577 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21", 2020-07-17T12:18:11.578 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.114341527Z", 2020-07-17T12:18:11.578 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.578 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.578 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.578 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.579 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.579 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_7IHshvdvzhTN_iaS63UlhhP", 2020-07-17T12:18:11.579 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.580 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.580 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.581 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.118095_117.1", 2020-07-17T12:18:11.581 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.581 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21", 2020-07-17T12:18:11.581 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.118095935Z", 2020-07-17T12:18:11.581 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.582 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.582 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.582 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.582 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.583 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_7IHshvdvzhTN_iaS63UlhhP", 2020-07-17T12:18:11.583 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.583 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.583 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.584 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.144536_119.1", 2020-07-17T12:18:11.585 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.585 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:11.585 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.144536481Z", 2020-07-17T12:18:11.585 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.585 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.585 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.586 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.586 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.586 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:11.586 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.587 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.587 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.587 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.587 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.588 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.148697_120.1", 2020-07-17T12:18:11.588 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.588 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:11.588 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.148697190Z", 2020-07-17T12:18:11.589 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.589 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.589 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.589 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.590 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.590 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:11.590 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.590 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.590 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.591 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.150760_121.1", 2020-07-17T12:18:11.592 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.592 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22", 2020-07-17T12:18:11.592 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.150760146Z", 2020-07-17T12:18:11.592 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.592 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.593 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.593 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.593 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.594 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gsPtnlqFRKAZ04Ktvzs95UN", 2020-07-17T12:18:11.595 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.595 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.595 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.596 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.155597_122.1", 2020-07-17T12:18:11.596 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.596 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22", 2020-07-17T12:18:11.597 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.155597648Z", 2020-07-17T12:18:11.597 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.597 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.597 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.597 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.598 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.598 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gsPtnlqFRKAZ04Ktvzs95UN", 2020-07-17T12:18:11.598 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.599 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.599 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.600 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.195741_123.1", 2020-07-17T12:18:11.600 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.600 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:11.600 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.195741697Z", 2020-07-17T12:18:11.600 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.601 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.601 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.601 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.601 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.602 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:11.602 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.602 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.602 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.602 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.603 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.603 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.200019_124.1", 2020-07-17T12:18:11.603 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.604 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:11.604 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.200019763Z", 2020-07-17T12:18:11.604 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.604 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.604 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.606 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.606 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.201967_125.1", 2020-07-17T12:18:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23", 2020-07-17T12:18:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.201967975Z", 2020-07-17T12:18:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.609 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_4EWRUD1pbHkY-FiDIwWfNaQ", 2020-07-17T12:18:11.609 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.609 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.609 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.610 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.610 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.205751_126.1", 2020-07-17T12:18:11.611 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.611 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23", 2020-07-17T12:18:11.611 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.205751555Z", 2020-07-17T12:18:11.611 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.611 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.611 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.612 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.612 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.612 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_4EWRUD1pbHkY-FiDIwWfNaQ", 2020-07-17T12:18:11.612 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.613 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.613 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.613 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.613 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.614 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.226616_127.1", 2020-07-17T12:18:11.614 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.614 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:11.614 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.226616279Z", 2020-07-17T12:18:11.615 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.615 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.615 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:11.615 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.615 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.616 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.616 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:11.616 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.616 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.617 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.617 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.617 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.617 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.231665_128.1", 2020-07-17T12:18:11.618 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.618 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:11.618 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.231665701Z", 2020-07-17T12:18:11.618 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.618 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.619 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:11.619 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.619 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.619 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:11.620 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.620 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.620 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.620 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.620 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.620 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.621 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.236032_130.1", 2020-07-17T12:18:11.621 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.621 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.236032642Z", 2020-07-17T12:18:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.623 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.623 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:11.623 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.623 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.625 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.240206_131.1", 2020-07-17T12:18:11.625 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.625 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:11.625 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.240206565Z", 2020-07-17T12:18:11.625 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.251296_132.1", 2020-07-17T12:18:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.629 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:11.629 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.251296772Z", 2020-07-17T12:18:11.629 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.629 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.629 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:11.629 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.630 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.630 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:11.630 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.632 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.262832_133.1", 2020-07-17T12:18:11.632 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.632 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:11.632 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.262832933Z", 2020-07-17T12:18:11.632 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.633 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.633 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:11.633 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.633 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.635 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.635 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.635 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.635 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.216266_162.1", 2020-07-17T12:18:11.635 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.636 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:11.636 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.216266874Z", 2020-07-17T12:18:11.636 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.636 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.636 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.636 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.637 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.637 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.637 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.638 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.638 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.638 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.638 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.639 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.639 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.220314_163.1", 2020-07-17T12:18:11.639 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.639 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:11.639 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.220314597Z", 2020-07-17T12:18:11.640 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.640 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.640 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.640 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.641 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.641 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.641 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.641 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.641 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.642 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.642 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.222492_164.1", 2020-07-17T12:18:11.643 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.643 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24", 2020-07-17T12:18:11.643 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.222492473Z", 2020-07-17T12:18:11.643 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.643 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.643 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.644 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.644 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.644 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1aTUtnO5v7_BzTHJ7dcWax9", 2020-07-17T12:18:11.644 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.645 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.645 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.646 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.226266_165.1", 2020-07-17T12:18:11.646 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.646 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24", 2020-07-17T12:18:11.646 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.226266565Z", 2020-07-17T12:18:11.647 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.647 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.647 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.647 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.647 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.648 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1aTUtnO5v7_BzTHJ7dcWax9", 2020-07-17T12:18:11.648 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.648 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.648 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.649 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.649 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.649 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.649 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.254586_166.1", 2020-07-17T12:18:11.649 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.650 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:11.650 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.254586031Z", 2020-07-17T12:18:11.650 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.650 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.650 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.651 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.651 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.651 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.651 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.652 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.652 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.652 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.652 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.653 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.259320_167.1", 2020-07-17T12:18:11.653 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.653 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:11.653 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.259320320Z", 2020-07-17T12:18:11.653 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.654 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.654 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.654 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.654 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.655 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.655 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.655 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.655 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.655 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.656 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.656 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.261985_168.1", 2020-07-17T12:18:11.656 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.657 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25", 2020-07-17T12:18:11.657 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.261985157Z", 2020-07-17T12:18:11.657 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.657 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.657 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.657 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.658 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.658 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_CJpPPaq5s3R_DLWK7_Dw6N9", 2020-07-17T12:18:11.658 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.659 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.659 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.660 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.268842_169.1", 2020-07-17T12:18:11.660 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.660 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25", 2020-07-17T12:18:11.660 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.268842944Z", 2020-07-17T12:18:11.660 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.660 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.661 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.661 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.661 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.661 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_CJpPPaq5s3R_DLWK7_Dw6N9", 2020-07-17T12:18:11.662 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.662 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.662 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.663 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.663 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.310784_170.1", 2020-07-17T12:18:11.663 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.663 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:11.664 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.310784639Z", 2020-07-17T12:18:11.664 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.664 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.664 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.664 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.665 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.665 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.665 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.665 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.666 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.666 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.666 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.666 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.667 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.319433_171.1", 2020-07-17T12:18:11.667 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.667 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:11.667 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.319433996Z", 2020-07-17T12:18:11.667 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.667 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.668 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.668 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.668 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.668 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.669 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.669 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.669 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.669 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.670 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.670 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.322090_172.1", 2020-07-17T12:18:11.670 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.670 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26", 2020-07-17T12:18:11.671 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.322090964Z", 2020-07-17T12:18:11.671 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.671 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.671 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.671 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.672 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.672 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_OI4YC_rV8-IvJU_ONTZMZ22", 2020-07-17T12:18:11.672 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.672 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.673 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.674 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.326100_173.1", 2020-07-17T12:18:11.674 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.674 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26", 2020-07-17T12:18:11.674 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.326100789Z", 2020-07-17T12:18:11.674 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.674 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.675 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.675 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.675 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.675 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_OI4YC_rV8-IvJU_ONTZMZ22", 2020-07-17T12:18:11.675 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.676 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.676 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.677 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.345795_174.1", 2020-07-17T12:18:11.677 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.677 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:11.677 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.345795397Z", 2020-07-17T12:18:11.678 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.678 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.678 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.678 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.679 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.679 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.679 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.679 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.679 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.680 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.680 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.680 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.351197_175.1", 2020-07-17T12:18:11.681 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.681 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:11.681 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.351197071Z", 2020-07-17T12:18:11.681 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.681 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.681 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.682 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.682 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.682 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.682 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.683 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.683 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.684 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.354716_176.1", 2020-07-17T12:18:11.684 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.684 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:11.684 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.354716106Z", 2020-07-17T12:18:11.684 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.685 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.685 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.685 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.685 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.686 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.686 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:11.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.686 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.686 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.687 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.687 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.360435_177.1", 2020-07-17T12:18:11.687 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.688 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:11.688 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.360435248Z", 2020-07-17T12:18:11.688 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.688 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.688 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.688 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.689 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.689 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:11.689 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:11.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.690 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.690 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.691 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.365795_178.1", 2020-07-17T12:18:11.691 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.691 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:11.691 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.365795374Z", 2020-07-17T12:18:11.691 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.692 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.692 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.692 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.692 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.693 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.693 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.693 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.693 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.693 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.694 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.372555_179.1", 2020-07-17T12:18:11.694 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.695 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:11.695 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.372555105Z", 2020-07-17T12:18:11.695 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.695 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.695 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.695 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.696 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.696 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.696 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.697 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.697 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.698 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.385196_180.1", 2020-07-17T12:18:11.698 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.698 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:11.698 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.385196882Z", 2020-07-17T12:18:11.698 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.699 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.699 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.699 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.699 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.699 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.700 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:11.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.700 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.700 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.701 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.393417_181.1", 2020-07-17T12:18:11.701 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.701 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:11.702 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.393417880Z", 2020-07-17T12:18:11.702 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.702 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.702 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.702 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.703 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.703 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:11.703 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:11.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.703 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.704 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.704 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.704 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.705 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.405697_182.1", 2020-07-17T12:18:11.705 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.705 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:11.705 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.405697153Z", 2020-07-17T12:18:11.705 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.705 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.706 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.706 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.706 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.706 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.707 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.707 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.707 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.707 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.707 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.708 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.415076_183.1", 2020-07-17T12:18:11.708 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.708 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:11.709 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.415076707Z", 2020-07-17T12:18:11.709 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.709 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.709 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.709 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.710 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.710 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.710 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.710 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.711 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.711 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.426554_184.1", 2020-07-17T12:18:11.712 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.712 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:11.712 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.426554025Z", 2020-07-17T12:18:11.712 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.712 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.713 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.713 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.713 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.713 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.713 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:11.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.714 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.714 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.714 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.714 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.715 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.715 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.433699_185.1", 2020-07-17T12:18:11.715 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.715 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:11.715 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.433699189Z", 2020-07-17T12:18:11.716 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.716 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.716 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.716 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.716 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.717 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:11.717 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:11.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.717 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.717 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.718 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.718 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.718 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.192609_186.1", 2020-07-17T12:18:11.718 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.719 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:11.719 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.192609023Z", 2020-07-17T12:18:11.719 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.719 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.719 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.720 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.720 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.720 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:11.720 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.721 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.721 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.721 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.721 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.722 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.197302_187.1", 2020-07-17T12:18:11.722 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.722 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:11.722 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.197302293Z", 2020-07-17T12:18:11.722 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.723 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.723 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.723 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.723 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.724 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:11.724 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.724 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.724 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.725 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.725 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.200146_188.1", 2020-07-17T12:18:11.725 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.725 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27", 2020-07-17T12:18:11.726 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.200146111Z", 2020-07-17T12:18:11.726 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.726 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.726 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.726 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.727 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.727 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_EAM3oic4BNiOLUongn3X0x8", 2020-07-17T12:18:11.727 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.727 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.728 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.728 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.729 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.204265_189.1", 2020-07-17T12:18:11.729 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.729 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27", 2020-07-17T12:18:11.729 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.204265093Z", 2020-07-17T12:18:11.729 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.729 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.730 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.730 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.730 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.730 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_EAM3oic4BNiOLUongn3X0x8", 2020-07-17T12:18:11.731 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.731 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.731 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.731 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.731 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.732 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.232604_190.1", 2020-07-17T12:18:11.732 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.732 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:11.733 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.232604283Z", 2020-07-17T12:18:11.733 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.733 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.733 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.733 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.734 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.734 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:11.734 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.734 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.735 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.735 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.236653_191.1", 2020-07-17T12:18:11.736 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.736 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:11.736 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.236653061Z", 2020-07-17T12:18:11.736 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.736 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.737 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.737 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.737 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.737 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:11.737 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.738 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.738 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.738 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.738 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.739 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.238308_192.1", 2020-07-17T12:18:11.739 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.739 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28", 2020-07-17T12:18:11.739 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.238308219Z", 2020-07-17T12:18:11.740 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.740 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.740 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.740 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.740 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.741 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_8tICQSSPhT4sUiRts2CGtXF", 2020-07-17T12:18:11.741 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.741 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.741 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.742 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.242121_193.1", 2020-07-17T12:18:11.742 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.743 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28", 2020-07-17T12:18:11.743 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.242121826Z", 2020-07-17T12:18:11.743 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.743 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.743 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.743 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.744 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.744 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_8tICQSSPhT4sUiRts2CGtXF", 2020-07-17T12:18:11.744 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.745 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.745 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.745 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.745 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.746 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.291427_194.1", 2020-07-17T12:18:11.746 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.746 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:11.746 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.291427038Z", 2020-07-17T12:18:11.746 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.747 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.747 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.747 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.747 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.747 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:11.748 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.748 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.748 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.749 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.296400_195.1", 2020-07-17T12:18:11.749 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.749 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:11.750 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.296400519Z", 2020-07-17T12:18:11.750 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.750 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.750 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.750 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.751 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.751 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:11.751 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.751 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.752 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.752 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.752 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.752 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.299100_196.1", 2020-07-17T12:18:11.753 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.753 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29", 2020-07-17T12:18:11.753 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.299100799Z", 2020-07-17T12:18:11.753 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.753 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.754 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.754 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.754 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.754 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oUHOV9TdQ-W-rj70_Cou9X5", 2020-07-17T12:18:11.754 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.755 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.755 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.755 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.755 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.756 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.304751_197.1", 2020-07-17T12:18:11.756 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.756 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29", 2020-07-17T12:18:11.756 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.304751236Z", 2020-07-17T12:18:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oUHOV9TdQ-W-rj70_Cou9X5", 2020-07-17T12:18:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.329636_198.1", 2020-07-17T12:18:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.329636096Z", 2020-07-17T12:18:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.762 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.762 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.762 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.762 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.763 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.334331_199.1", 2020-07-17T12:18:11.763 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.763 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:11.764 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.334331520Z", 2020-07-17T12:18:11.764 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.764 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.765 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:11.765 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.765 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.766 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:11.766 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.766 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.388479_200.1", 2020-07-17T12:18:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.388479756Z", 2020-07-17T12:18:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:11.770 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.770 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.770 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.771 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.398272_201.1", 2020-07-17T12:18:11.771 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.771 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:11.772 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.398272953Z", 2020-07-17T12:18:11.772 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.772 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.772 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:11.772 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.773 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.773 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:11.773 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.773 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.774 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.775 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.463878_202.1", 2020-07-17T12:18:11.775 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.775 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:11.775 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.463878801Z", 2020-07-17T12:18:11.775 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.775 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.776 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:11.776 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.776 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.776 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:11.777 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.777 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.777 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.777 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.777 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.778 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.474476_203.1", 2020-07-17T12:18:11.778 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.778 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:11.779 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.474476699Z", 2020-07-17T12:18:11.779 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.779 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.779 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:11.779 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.780 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.780 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:11.780 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.780 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.780 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.781 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.781 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.543823_204.1", 2020-07-17T12:18:11.782 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.782 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:11.782 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.543823468Z", 2020-07-17T12:18:11.782 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.782 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.782 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.783 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.783 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.783 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:11.783 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.784 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.784 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.784 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.784 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.785 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.548616_205.1", 2020-07-17T12:18:11.785 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.785 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:11.785 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.548616662Z", 2020-07-17T12:18:11.786 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.786 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.786 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.786 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.786 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.787 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:11.787 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.787 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.787 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.788 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.788 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.788 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.550563_206.1", 2020-07-17T12:18:11.788 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.789 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30", 2020-07-17T12:18:11.789 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.550563904Z", 2020-07-17T12:18:11.789 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.789 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.789 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.789 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.790 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.790 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_lJ2S3jRGkcD2jZEcb0bU-NY", 2020-07-17T12:18:11.790 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.791 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.791 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.791 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.791 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.792 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.555228_207.1", 2020-07-17T12:18:11.792 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.792 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30", 2020-07-17T12:18:11.792 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.555228975Z", 2020-07-17T12:18:11.792 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.793 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.793 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.793 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.793 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.793 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_lJ2S3jRGkcD2jZEcb0bU-NY", 2020-07-17T12:18:11.794 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.794 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.794 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.803 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.804 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.804 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.579434_208.1", 2020-07-17T12:18:11.804 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.804 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:11.805 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.579434456Z", 2020-07-17T12:18:11.805 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.805 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.805 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.805 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.806 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.806 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:11.806 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.806 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.807 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.807 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.807 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.808 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.583820_209.1", 2020-07-17T12:18:11.808 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.808 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:11.808 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.583820075Z", 2020-07-17T12:18:11.808 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.808 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.809 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.809 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.809 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.809 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:11.809 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.810 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.810 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.810 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.810 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.811 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.585899_210.1", 2020-07-17T12:18:11.811 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.811 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31", 2020-07-17T12:18:11.811 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.585899826Z", 2020-07-17T12:18:11.812 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.812 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.812 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.812 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.813 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.813 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_ln1eOUq8lDkVI4IpQvkNAi4", 2020-07-17T12:18:11.813 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.813 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.813 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.814 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.814 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.814 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.589417_211.1", 2020-07-17T12:18:11.814 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.815 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31", 2020-07-17T12:18:11.815 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.589417398Z", 2020-07-17T12:18:11.815 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.815 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.815 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.816 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.816 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.816 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_ln1eOUq8lDkVI4IpQvkNAi4", 2020-07-17T12:18:11.816 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.817 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.817 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.817 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.817 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.818 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.638248_212.1", 2020-07-17T12:18:11.818 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.818 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:11.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.638248979Z", 2020-07-17T12:18:11.818 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.819 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.819 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.819 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.819 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.820 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:11.820 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.820 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.820 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.821 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.642956_213.1", 2020-07-17T12:18:11.821 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.821 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:11.822 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.642956157Z", 2020-07-17T12:18:11.822 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.822 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.822 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.822 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.823 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.823 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:11.823 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.823 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.824 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.825 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.644834_214.1", 2020-07-17T12:18:11.825 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.825 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32", 2020-07-17T12:18:11.825 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.644834814Z", 2020-07-17T12:18:11.825 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.826 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.826 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.826 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.826 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.826 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_CTGkGtU8NTFdGu11wP4HBlE", 2020-07-17T12:18:11.827 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.827 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.827 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.827 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.827 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.828 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.828 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.649548_215.1", 2020-07-17T12:18:11.828 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.828 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32", 2020-07-17T12:18:11.829 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.649548275Z", 2020-07-17T12:18:11.829 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.829 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.829 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.829 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.830 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.830 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_CTGkGtU8NTFdGu11wP4HBlE", 2020-07-17T12:18:11.830 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.830 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.831 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.831 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.691484_216.1", 2020-07-17T12:18:11.832 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.832 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:11.832 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.691484060Z", 2020-07-17T12:18:11.832 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.832 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.832 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:11.833 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.833 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.833 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:11.833 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.834 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.834 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.835 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.695812_217.1", 2020-07-17T12:18:11.835 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.835 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:11.835 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.695812573Z", 2020-07-17T12:18:11.836 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.836 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.836 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:11.836 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.836 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.837 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:11.837 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.837 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.837 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.838 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.838 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.752305_218.1", 2020-07-17T12:18:11.838 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.839 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:11.839 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.752305251Z", 2020-07-17T12:18:11.839 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.839 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.839 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:11.839 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.840 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.840 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:11.840 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.841 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.841 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.842 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.759930_219.1", 2020-07-17T12:18:11.842 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.842 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:11.842 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.759930328Z", 2020-07-17T12:18:11.842 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.843 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.843 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:11.843 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.843 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.843 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:11.844 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.844 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.844 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.844 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.845 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.819822_220.1", 2020-07-17T12:18:11.845 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.845 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:11.846 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.819822109Z", 2020-07-17T12:18:11.846 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.846 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.846 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:11.846 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.847 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.847 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:11.847 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.847 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.848 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.848 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.848 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.848 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.849 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.829887_221.1", 2020-07-17T12:18:11.849 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.849 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:11.849 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.829887197Z", 2020-07-17T12:18:11.849 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.849 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.850 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:11.850 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.850 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.850 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:11.851 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.851 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.851 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.852 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.897711_222.1", 2020-07-17T12:18:11.852 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.852 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:11.853 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.897711412Z", 2020-07-17T12:18:11.853 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.853 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.853 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.853 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.854 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.854 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:11.854 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.854 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.854 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.855 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.855 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.855 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.901139_223.1", 2020-07-17T12:18:11.856 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.856 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:11.856 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.901139991Z", 2020-07-17T12:18:11.856 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.856 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.857 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.857 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.857 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.857 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:11.857 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.858 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.858 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.859 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.902679_224.1", 2020-07-17T12:18:11.859 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.859 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33", 2020-07-17T12:18:11.859 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.902679722Z", 2020-07-17T12:18:11.860 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.860 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.860 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.860 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.860 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.861 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_UxWO-19hsf0zfhgE2d7C0Bs", 2020-07-17T12:18:11.861 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.861 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.861 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.862 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.862 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.862 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.906270_225.1", 2020-07-17T12:18:11.862 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.863 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33", 2020-07-17T12:18:11.863 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.906270781Z", 2020-07-17T12:18:11.863 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.863 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.863 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.864 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.864 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.864 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_UxWO-19hsf0zfhgE2d7C0Bs", 2020-07-17T12:18:11.864 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.865 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.865 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.865 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.865 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.866 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.866 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.932251_226.1", 2020-07-17T12:18:11.866 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.866 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:11.866 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.932251093Z", 2020-07-17T12:18:11.866 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.867 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.867 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.867 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.867 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.868 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:11.868 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.868 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.868 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.868 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.869 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.869 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.869 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.937747_227.1", 2020-07-17T12:18:11.869 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.870 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:11.870 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.937747244Z", 2020-07-17T12:18:11.870 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.870 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.870 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.870 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.871 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.871 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:11.871 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.872 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.872 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.872 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.872 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.873 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.939489_228.1", 2020-07-17T12:18:11.873 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.873 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34", 2020-07-17T12:18:11.873 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.939489887Z", 2020-07-17T12:18:11.873 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.874 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.874 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.874 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.874 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.874 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oKhT8xShmiPT7_3DwLLaiZf", 2020-07-17T12:18:11.875 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.875 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.875 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.875 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.875 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.876 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.942587_229.1", 2020-07-17T12:18:11.876 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.876 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34", 2020-07-17T12:18:11.877 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.942587543Z", 2020-07-17T12:18:11.877 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.877 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.877 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.877 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.878 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.878 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oKhT8xShmiPT7_3DwLLaiZf", 2020-07-17T12:18:11.878 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.878 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.879 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.879 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.879 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.879 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.977552_230.1", 2020-07-17T12:18:11.880 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.880 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:11.880 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.977552760Z", 2020-07-17T12:18:11.880 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.880 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.881 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.881 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.881 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.881 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:11.881 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.882 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.882 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.883 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.982605_231.1", 2020-07-17T12:18:11.883 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.883 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:11.883 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.982605704Z", 2020-07-17T12:18:11.884 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.884 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.884 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.884 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.884 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.885 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:11.885 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.885 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.885 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.886 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.985248_232.1", 2020-07-17T12:18:11.887 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.887 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35", 2020-07-17T12:18:11.887 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.985248699Z", 2020-07-17T12:18:11.887 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.887 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.887 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.888 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.888 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.888 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_vxH0cpq2NUCS2-F5AwiLlGy", 2020-07-17T12:18:11.888 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.889 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.889 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.889 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.889 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.890 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.989554_233.1", 2020-07-17T12:18:11.890 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.890 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35", 2020-07-17T12:18:11.890 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.989554870Z", 2020-07-17T12:18:11.891 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.891 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.891 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.891 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.891 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.892 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_vxH0cpq2NUCS2-F5AwiLlGy", 2020-07-17T12:18:11.892 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.892 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.892 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.893 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988149.009800_234.1", 2020-07-17T12:18:11.894 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.894 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:11.894 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:49.009800265Z", 2020-07-17T12:18:11.894 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.894 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.894 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:11.895 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.895 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.895 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:11.895 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.896 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.896 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.897 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988149.014525_235.1", 2020-07-17T12:18:11.897 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.897 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:11.897 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:49.014525305Z", 2020-07-17T12:18:11.898 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.898 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.898 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:11.898 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.898 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.899 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:11.899 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.899 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.899 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.900 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.900 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.900 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988150.064957_236.1", 2020-07-17T12:18:11.900 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.901 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:11.901 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:50.064957041Z", 2020-07-17T12:18:11.901 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.901 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.901 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:11.902 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.902 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.902 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:11.902 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.903 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.903 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.904 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988150.071666_237.1", 2020-07-17T12:18:11.904 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.904 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:11.904 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:50.071666653Z", 2020-07-17T12:18:11.904 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.905 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.905 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:11.905 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.905 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.906 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:11.906 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.906 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.906 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.906 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.907 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.907 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.128899_238.1", 2020-07-17T12:18:11.907 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.907 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:11.908 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.128899145Z", 2020-07-17T12:18:11.908 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.908 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.908 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:11.908 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.909 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.909 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:11.909 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.909 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.910 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.910 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.911 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.136044_239.1", 2020-07-17T12:18:11.911 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.911 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:11.911 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.136044802Z", 2020-07-17T12:18:11.911 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.911 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.912 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:11.912 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.912 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.912 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:11.913 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:11.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.913 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.913 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.913 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.914 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.223773_240.1", 2020-07-17T12:18:11.914 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.914 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-36:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.36", 2020-07-17T12:18:11.915 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.223773094Z", 2020-07-17T12:18:11.915 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.915 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.915 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.915 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.916 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.916 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jdX6lkwwXVktMAEuRsLdL9f", 2020-07-17T12:18:11.916 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.916 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.917 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.917 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.917 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.227459_241.1", 2020-07-17T12:18:11.918 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.918 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-36:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.36", 2020-07-17T12:18:11.918 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.227459296Z", 2020-07-17T12:18:11.918 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.918 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.919 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.919 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.919 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.919 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jdX6lkwwXVktMAEuRsLdL9f", 2020-07-17T12:18:11.919 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.920 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.920 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.920 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.920 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.921 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.229174_242.1", 2020-07-17T12:18:11.921 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.921 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-36", 2020-07-17T12:18:11.921 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.229174371Z", 2020-07-17T12:18:11.922 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.922 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.922 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.922 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.923 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.923 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_qgrqKYBWaNJyRRn3Vjx0wVN", 2020-07-17T12:18:11.923 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.923 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.923 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.924 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.924 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.924 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.232729_243.1", 2020-07-17T12:18:11.925 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.925 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-36", 2020-07-17T12:18:11.925 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.232729435Z", 2020-07-17T12:18:11.925 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.925 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.925 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.926 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.926 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.926 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_qgrqKYBWaNJyRRn3Vjx0wVN", 2020-07-17T12:18:11.926 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.927 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.927 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.927 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.927 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.928 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.928 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.269955_244.1", 2020-07-17T12:18:11.928 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.928 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-37:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.37", 2020-07-17T12:18:11.928 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.269955795Z", 2020-07-17T12:18:11.929 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.929 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.929 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.929 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.930 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.930 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_c-tGp8YfI6B1yBpFUhIj44i", 2020-07-17T12:18:11.930 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.930 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.930 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.931 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.931 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.931 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.275803_245.1", 2020-07-17T12:18:11.932 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.932 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-37:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.37", 2020-07-17T12:18:11.932 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.275803241Z", 2020-07-17T12:18:11.932 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.932 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.932 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.933 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.933 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.933 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_c-tGp8YfI6B1yBpFUhIj44i", 2020-07-17T12:18:11.933 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.934 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.934 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.935 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.278648_246.1", 2020-07-17T12:18:11.935 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.935 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-37", 2020-07-17T12:18:11.935 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.278648356Z", 2020-07-17T12:18:11.935 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.936 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.936 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.936 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.936 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.937 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_9F_FGhwZ3EnXzZglCBoTwNA", 2020-07-17T12:18:11.937 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.937 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.937 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.937 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.938 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.938 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.285189_247.1", 2020-07-17T12:18:11.938 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.939 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-37", 2020-07-17T12:18:11.939 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.285189703Z", 2020-07-17T12:18:11.939 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.939 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.939 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.939 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.940 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.940 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_9F_FGhwZ3EnXzZglCBoTwNA", 2020-07-17T12:18:11.940 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.940 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.941 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.941 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.941 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.941 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.942 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.334180_248.1", 2020-07-17T12:18:11.942 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.942 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-38:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.38", 2020-07-17T12:18:11.942 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.334180801Z", 2020-07-17T12:18:11.942 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.943 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.943 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.943 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.943 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.943 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_hVnb1ngvk17nkYrOuLaCqwM", 2020-07-17T12:18:11.944 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.944 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.944 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.945 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.945 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.337583_249.1", 2020-07-17T12:18:11.945 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.945 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-38:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.38", 2020-07-17T12:18:11.946 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.337583158Z", 2020-07-17T12:18:11.946 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.946 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.946 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.946 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.947 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.947 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_hVnb1ngvk17nkYrOuLaCqwM", 2020-07-17T12:18:11.947 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.947 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.948 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.949 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.339086_250.1", 2020-07-17T12:18:11.949 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.949 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-38", 2020-07-17T12:18:11.949 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.339086250Z", 2020-07-17T12:18:11.949 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.949 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.950 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.950 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.950 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.950 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_3EGy4bfasMLEwsmYrKxh2v5", 2020-07-17T12:18:11.951 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.951 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.951 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.952 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.952 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.342294_251.1", 2020-07-17T12:18:11.952 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.952 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-38", 2020-07-17T12:18:11.953 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.342294371Z", 2020-07-17T12:18:11.953 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.953 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.953 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.953 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.954 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.954 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_3EGy4bfasMLEwsmYrKxh2v5", 2020-07-17T12:18:11.954 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.954 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.955 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.955 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.955 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.955 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.955 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.949522_252.1", 2020-07-17T12:18:11.956 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.956 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-39:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.39", 2020-07-17T12:18:11.956 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.949522585Z", 2020-07-17T12:18:11.956 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.956 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.957 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.957 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.957 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.957 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gsVvIhPoNrpxq4YxIZghI20", 2020-07-17T12:18:11.958 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.958 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.958 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.958 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.958 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.959 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.956039_253.1", 2020-07-17T12:18:11.959 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.959 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-39:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.39", 2020-07-17T12:18:11.960 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.956039623Z", 2020-07-17T12:18:11.960 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.960 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.960 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.960 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.961 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.961 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gsVvIhPoNrpxq4YxIZghI20", 2020-07-17T12:18:11.961 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.961 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.962 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.962 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.962 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.962 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.962 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.961169_254.1", 2020-07-17T12:18:11.963 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.963 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-39", 2020-07-17T12:18:11.963 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.961169072Z", 2020-07-17T12:18:11.963 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.963 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.964 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.964 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.964 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.964 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jHFRYvjwFaeKft8jqkFw9f6", 2020-07-17T12:18:11.964 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.965 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.965 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.966 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.966 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.965532_255.1", 2020-07-17T12:18:11.966 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.966 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-39", 2020-07-17T12:18:11.966 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.965532718Z", 2020-07-17T12:18:11.967 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.967 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.967 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.967 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jHFRYvjwFaeKft8jqkFw9f6", 2020-07-17T12:18:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.969 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.969 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.969 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.996911_256.1", 2020-07-17T12:18:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-40:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.40", 2020-07-17T12:18:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.996911090Z", 2020-07-17T12:18:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zd9daTaLaaQPrwIpjVSY9v4", 2020-07-17T12:18:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.973 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.001607_257.1", 2020-07-17T12:18:11.973 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.973 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-40:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.40", 2020-07-17T12:18:11.973 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.001607160Z", 2020-07-17T12:18:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zd9daTaLaaQPrwIpjVSY9v4", 2020-07-17T12:18:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.004358_258.1", 2020-07-17T12:18:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-40", 2020-07-17T12:18:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.004358398Z", 2020-07-17T12:18:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Apkycxzmtorr0IIi3gzU-Qm", 2020-07-17T12:18:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.979 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.979 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.979 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.979 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.980 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.009492_259.1", 2020-07-17T12:18:11.980 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.980 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-40", 2020-07-17T12:18:11.980 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.009492644Z", 2020-07-17T12:18:11.981 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.981 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.981 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.981 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.981 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.982 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Apkycxzmtorr0IIi3gzU-Qm", 2020-07-17T12:18:11.982 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.982 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.982 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.048556_260.1", 2020-07-17T12:18:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.984 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-41:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.41", 2020-07-17T12:18:11.984 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.048556572Z", 2020-07-17T12:18:11.984 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.984 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.984 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NntBaMu1tW2iuZqebtOm9OP", 2020-07-17T12:18:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.053032_261.1", 2020-07-17T12:18:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-41:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.41", 2020-07-17T12:18:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.053032571Z", 2020-07-17T12:18:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NntBaMu1tW2iuZqebtOm9OP", 2020-07-17T12:18:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.055326_262.1", 2020-07-17T12:18:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-41", 2020-07-17T12:18:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.055326711Z", 2020-07-17T12:18:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.992 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.992 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_W8htjS8YlHsEASKVqW9GGew", 2020-07-17T12:18:11.992 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.994 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.060681_263.1", 2020-07-17T12:18:11.994 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:11.994 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-41", 2020-07-17T12:18:11.994 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.060681987Z", 2020-07-17T12:18:11.994 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:11.994 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_W8htjS8YlHsEASKVqW9GGew", 2020-07-17T12:18:11.996 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:11.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.996 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:11.996 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:11.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.997 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:11.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:11.997 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:11.997 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.299855_42.1", 2020-07-17T12:18:11.997 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:11.997 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-1:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.1", 2020-07-17T12:18:12.005 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.299855768Z", 2020-07-17T12:18:12.005 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.005 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.005 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.006 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.006 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.006 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fCYmrES933MNW-ZftW97tAH", 2020-07-17T12:18:12.006 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.007 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.007 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.007 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.007 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.008 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.318362_43.1", 2020-07-17T12:18:12.008 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.008 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-1:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.1", 2020-07-17T12:18:12.008 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.318362600Z", 2020-07-17T12:18:12.009 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.009 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.009 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.009 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.010 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.010 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fCYmrES933MNW-ZftW97tAH", 2020-07-17T12:18:12.010 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.010 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.010 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.011 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.322473_44.1", 2020-07-17T12:18:12.012 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.012 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-1", 2020-07-17T12:18:12.012 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.322473615Z", 2020-07-17T12:18:12.012 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.012 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.012 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.013 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.013 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.013 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_N3sIyaABoPKhvVJDTubfANC", 2020-07-17T12:18:12.013 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.014 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.014 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.014 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.014 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.015 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.015 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.326365_45.1", 2020-07-17T12:18:12.015 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.015 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-1", 2020-07-17T12:18:12.015 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.326365120Z", 2020-07-17T12:18:12.016 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.016 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.016 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.016 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.017 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.017 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_N3sIyaABoPKhvVJDTubfANC", 2020-07-17T12:18:12.017 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.017 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.018 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.018 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.018 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.018 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.019 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.371051_46.1", 2020-07-17T12:18:12.019 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.019 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-2:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.2", 2020-07-17T12:18:12.019 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.371051231Z", 2020-07-17T12:18:12.019 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.019 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.020 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.020 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.020 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.020 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_bpM5KcuqzRHhKrpzuuAWOQK", 2020-07-17T12:18:12.021 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.021 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.021 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.021 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.021 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.022 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.376155_47.1", 2020-07-17T12:18:12.022 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.022 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-2:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.2", 2020-07-17T12:18:12.023 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.376155577Z", 2020-07-17T12:18:12.023 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.023 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.023 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.023 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.024 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.024 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_bpM5KcuqzRHhKrpzuuAWOQK", 2020-07-17T12:18:12.024 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.024 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.025 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.026 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.378460_48.1", 2020-07-17T12:18:12.026 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.026 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-2", 2020-07-17T12:18:12.026 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.378460095Z", 2020-07-17T12:18:12.026 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.026 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.027 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.027 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.027 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.027 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_o0AcCUXo7hxp6mur0vpSEA6", 2020-07-17T12:18:12.028 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.028 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.028 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.028 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.028 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.029 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.382370_49.1", 2020-07-17T12:18:12.029 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.029 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-2", 2020-07-17T12:18:12.030 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.382370235Z", 2020-07-17T12:18:12.030 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.030 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.030 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.030 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.031 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.031 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_o0AcCUXo7hxp6mur0vpSEA6", 2020-07-17T12:18:12.031 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.031 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.032 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.033 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987598.171972_50.1", 2020-07-17T12:18:12.033 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.033 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-3:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.3", 2020-07-17T12:18:12.033 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.171972137Z", 2020-07-17T12:18:12.033 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.033 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.034 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.034 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.034 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.034 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jXpmpU1bMKc6cP4eCcdC0E5", 2020-07-17T12:18:12.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.035 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.035 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.035 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.035 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.036 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987598.175455_51.1", 2020-07-17T12:18:12.036 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.036 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-3:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.3", 2020-07-17T12:18:12.037 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.175455638Z", 2020-07-17T12:18:12.037 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.037 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.037 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.037 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.038 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.038 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jXpmpU1bMKc6cP4eCcdC0E5", 2020-07-17T12:18:12.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.038 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.039 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.039 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.039 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987598.177003_52.1", 2020-07-17T12:18:12.040 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.040 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-3", 2020-07-17T12:18:12.040 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.177003803Z", 2020-07-17T12:18:12.040 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.040 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.041 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.041 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.041 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.041 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_-b65PCt1-zecno94kbqdZQV", 2020-07-17T12:18:12.041 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.042 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.042 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.042 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.042 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.043 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987598.180439_53.1", 2020-07-17T12:18:12.043 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.043 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-3", 2020-07-17T12:18:12.043 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.180439200Z", 2020-07-17T12:18:12.044 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.044 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.044 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.044 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.044 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.045 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_-b65PCt1-zecno94kbqdZQV", 2020-07-17T12:18:12.045 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.045 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.045 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.046 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.046 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.046 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.433011_54.1", 2020-07-17T12:18:12.046 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.047 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-4:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.4", 2020-07-17T12:18:12.047 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.433011959Z", 2020-07-17T12:18:12.047 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.047 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.047 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.048 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.048 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.048 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__ea-Segy13dxgNiJPXZ7Zd9", 2020-07-17T12:18:12.048 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.049 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.049 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.050 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.438333_55.1", 2020-07-17T12:18:12.050 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.050 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-4:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.4", 2020-07-17T12:18:12.050 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.438333452Z", 2020-07-17T12:18:12.050 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.051 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.051 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.051 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.051 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.051 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__ea-Segy13dxgNiJPXZ7Zd9", 2020-07-17T12:18:12.052 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.052 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.052 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.053 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.440677_56.1", 2020-07-17T12:18:12.053 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.053 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-4", 2020-07-17T12:18:12.054 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.440677898Z", 2020-07-17T12:18:12.054 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.054 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.054 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.054 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.055 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.055 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_WRyLUNBmhHEnabp7o-DhlbH", 2020-07-17T12:18:12.055 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.055 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.056 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.057 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.445531_57.1", 2020-07-17T12:18:12.057 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.057 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-4", 2020-07-17T12:18:12.057 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.445531158Z", 2020-07-17T12:18:12.057 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.057 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.058 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.058 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.058 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.058 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_WRyLUNBmhHEnabp7o-DhlbH", 2020-07-17T12:18:12.059 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.059 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.059 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.060 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.468350_58.1", 2020-07-17T12:18:12.060 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.060 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-5:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.5", 2020-07-17T12:18:12.061 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.468350431Z", 2020-07-17T12:18:12.061 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.061 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.061 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.061 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.062 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.062 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_FMcLIdx8RNwH2qOVrLRBKLI", 2020-07-17T12:18:12.062 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.062 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.063 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.063 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.064 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.472556_59.1", 2020-07-17T12:18:12.064 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.064 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-5:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.5", 2020-07-17T12:18:12.064 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.472556928Z", 2020-07-17T12:18:12.064 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.064 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.065 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.065 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.065 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.065 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_FMcLIdx8RNwH2qOVrLRBKLI", 2020-07-17T12:18:12.066 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.066 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.066 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.067 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.474186_60.1", 2020-07-17T12:18:12.067 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.067 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-5", 2020-07-17T12:18:12.068 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.474186224Z", 2020-07-17T12:18:12.068 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.068 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.068 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.068 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.069 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.069 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_7oOlSD0kLzDgbQh79FmiwVN", 2020-07-17T12:18:12.069 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.069 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.069 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.070 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.070 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.477662_61.1", 2020-07-17T12:18:12.071 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.071 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-5", 2020-07-17T12:18:12.071 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.477662122Z", 2020-07-17T12:18:12.071 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.071 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.071 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.072 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.072 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.072 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_7oOlSD0kLzDgbQh79FmiwVN", 2020-07-17T12:18:12.072 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.073 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.073 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.074 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.518094_62.1", 2020-07-17T12:18:12.074 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.074 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-6:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.6", 2020-07-17T12:18:12.074 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.518094230Z", 2020-07-17T12:18:12.075 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.075 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.075 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.075 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.075 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.076 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_98i-mz1lveZeu2xWlqfjP8v", 2020-07-17T12:18:12.076 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.076 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.076 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.076 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.077 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.525483_63.1", 2020-07-17T12:18:12.077 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.078 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-6:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.6", 2020-07-17T12:18:12.078 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.525483008Z", 2020-07-17T12:18:12.078 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.078 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.078 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.079 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.079 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.079 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_98i-mz1lveZeu2xWlqfjP8v", 2020-07-17T12:18:12.079 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.080 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.080 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.081 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.527486_64.1", 2020-07-17T12:18:12.081 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.081 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-6", 2020-07-17T12:18:12.081 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.527486008Z", 2020-07-17T12:18:12.081 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.082 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.082 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.082 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.083 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6KmjjH6ijuVTEeEq7jAXi7H", 2020-07-17T12:18:12.083 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.083 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.083 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.533990_65.1", 2020-07-17T12:18:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-6", 2020-07-17T12:18:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.533990618Z", 2020-07-17T12:18:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6KmjjH6ijuVTEeEq7jAXi7H", 2020-07-17T12:18:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.314142_66.1", 2020-07-17T12:18:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.7", 2020-07-17T12:18:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.314142380Z", 2020-07-17T12:18:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.089 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.089 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.089 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.089 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_3p7wlqKX2jIqNSa48PpGSPB", 2020-07-17T12:18:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.091 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.091 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.321643_67.1", 2020-07-17T12:18:12.091 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.091 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.7", 2020-07-17T12:18:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.321643612Z", 2020-07-17T12:18:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.093 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.093 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_3p7wlqKX2jIqNSa48PpGSPB", 2020-07-17T12:18:12.093 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.093 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.324252_68.1", 2020-07-17T12:18:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7", 2020-07-17T12:18:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.324252955Z", 2020-07-17T12:18:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.096 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.096 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.096 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:12.096 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.097 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.097 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.097 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.097 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.098 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.098 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.329888_69.1", 2020-07-17T12:18:12.098 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.098 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7", 2020-07-17T12:18:12.098 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.329888896Z", 2020-07-17T12:18:12.099 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.099 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.099 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.099 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.099 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.100 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:12.100 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.100 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.100 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.101 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.360390_70.1", 2020-07-17T12:18:12.102 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.102 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.8", 2020-07-17T12:18:12.102 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.360390239Z", 2020-07-17T12:18:12.102 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.102 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.102 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.103 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.103 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.103 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r09s2aK_hqiAA214GJtpxF1", 2020-07-17T12:18:12.103 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.104 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.104 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.105 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.105 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.364581_71.1", 2020-07-17T12:18:12.105 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.105 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.8", 2020-07-17T12:18:12.105 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.364581063Z", 2020-07-17T12:18:12.106 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.106 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.106 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.106 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.107 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.107 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r09s2aK_hqiAA214GJtpxF1", 2020-07-17T12:18:12.107 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.107 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.107 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.108 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.108 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.369486_72.1", 2020-07-17T12:18:12.109 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.109 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8", 2020-07-17T12:18:12.109 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.369486518Z", 2020-07-17T12:18:12.109 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.109 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.109 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.110 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.110 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.110 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:12.110 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.111 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.111 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.112 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.373949_73.1", 2020-07-17T12:18:12.112 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.112 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8", 2020-07-17T12:18:12.112 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.373949060Z", 2020-07-17T12:18:12.112 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.113 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.113 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.113 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.113 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.114 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:12.114 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.114 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.114 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.115 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.424920_75.1", 2020-07-17T12:18:12.115 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.116 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.9", 2020-07-17T12:18:12.116 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.424920583Z", 2020-07-17T12:18:12.116 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.116 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.116 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.117 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.117 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.117 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_znrNp-QgDthcyR2ClROORQe", 2020-07-17T12:18:12.117 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.118 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.118 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.119 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.119 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.431645_76.1", 2020-07-17T12:18:12.119 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.119 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.9", 2020-07-17T12:18:12.119 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.431645809Z", 2020-07-17T12:18:12.119 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.120 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.120 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.120 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.120 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.120 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_znrNp-QgDthcyR2ClROORQe", 2020-07-17T12:18:12.121 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.121 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.121 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.121 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.434319_77.1", 2020-07-17T12:18:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9", 2020-07-17T12:18:12.123 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.434319003Z", 2020-07-17T12:18:12.123 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.123 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.123 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.123 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.439076_78.1", 2020-07-17T12:18:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9", 2020-07-17T12:18:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.439076836Z", 2020-07-17T12:18:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.701989_80.1", 2020-07-17T12:18:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7", 2020-07-17T12:18:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.701989372Z", 2020-07-17T12:18:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.131 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:12.131 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:12.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.131 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.131 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "remove" 2020-07-17T12:18:12.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.132 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.708007_81.1", 2020-07-17T12:18:12.132 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.133 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7", 2020-07-17T12:18:12.133 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.708007772Z", 2020-07-17T12:18:12.133 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.133 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.133 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.136 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.781097_82.1", 2020-07-17T12:18:12.136 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.136 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8", 2020-07-17T12:18:12.136 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.781097190Z", 2020-07-17T12:18:12.136 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.137 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.137 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:12.137 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.137 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.137 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:12.138 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:12.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.138 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.138 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "remove" 2020-07-17T12:18:12.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.139 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.786755_83.1", 2020-07-17T12:18:12.139 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.139 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8", 2020-07-17T12:18:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.786755007Z", 2020-07-17T12:18:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.141 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.142 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:12.142 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:12.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.142 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.889801_85.1", 2020-07-17T12:18:12.144 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.144 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9", 2020-07-17T12:18:12.144 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.889801769Z", 2020-07-17T12:18:12.144 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.144 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:12.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.146 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.146 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "remove" 2020-07-17T12:18:12.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.895710_86.1", 2020-07-17T12:18:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9", 2020-07-17T12:18:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710719Z", 2020-07-17T12:18:12.148 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:12.148 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:12.148 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:12.148 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:12.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:12.150 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:12.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:12.150 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:18:12.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:12.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:12.153 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 34954458-202e-48c3-864b-c00b15f41dd5 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:12.191 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:12.191 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:12.376 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987471.882144_44.1", 2020-07-17T12:18:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "user", 2020-07-17T12:18:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rgw-multisite-test-user", 2020-07-17T12:18:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.882144008Z", 2020-07-17T12:18:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_uJ2kpFkaIMR3kSeSIGfEB3T", 2020-07-17T12:18:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987471.890977_45.1", 2020-07-17T12:18:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "user", 2020-07-17T12:18:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rgw-multisite-test-user", 2020-07-17T12:18:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:04:31.890977275Z", 2020-07-17T12:18:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_uJ2kpFkaIMR3kSeSIGfEB3T", 2020-07-17T12:18:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.172466_48.1", 2020-07-17T12:18:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-10:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.10", 2020-07-17T12:18:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.172466793Z", 2020-07-17T12:18:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_mG7u6kOIVSSgWYLweK5uzvb", 2020-07-17T12:18:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.177556_49.1", 2020-07-17T12:18:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-10:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.10", 2020-07-17T12:18:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.177556387Z", 2020-07-17T12:18:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_mG7u6kOIVSSgWYLweK5uzvb", 2020-07-17T12:18:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.182179_50.1", 2020-07-17T12:18:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-10", 2020-07-17T12:18:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.182179376Z", 2020-07-17T12:18:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_yuMwADwMvlFMi5Y6H00iObr", 2020-07-17T12:18:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.186476_51.1", 2020-07-17T12:18:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-10", 2020-07-17T12:18:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.186476843Z", 2020-07-17T12:18:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_yuMwADwMvlFMi5Y6H00iObr", 2020-07-17T12:18:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.216713_52.1", 2020-07-17T12:18:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-11:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.11", 2020-07-17T12:18:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.216713690Z", 2020-07-17T12:18:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gq5MHLpwjP0npYemQW4217g", 2020-07-17T12:18:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.222092_53.1", 2020-07-17T12:18:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-11:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.11", 2020-07-17T12:18:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.222092496Z", 2020-07-17T12:18:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gq5MHLpwjP0npYemQW4217g", 2020-07-17T12:18:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.223732_54.1", 2020-07-17T12:18:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-11", 2020-07-17T12:18:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.223732073Z", 2020-07-17T12:18:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_hmDQZxZXEQLZ3GX5-Ia1DXm", 2020-07-17T12:18:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.227773_55.1", 2020-07-17T12:18:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-11", 2020-07-17T12:18:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.227773848Z", 2020-07-17T12:18:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_hmDQZxZXEQLZ3GX5-Ia1DXm", 2020-07-17T12:18:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.263759_56.1", 2020-07-17T12:18:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-12:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.12", 2020-07-17T12:18:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.263759779Z", 2020-07-17T12:18:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_kJPn3awfkoAAv2YoILjX1YX", 2020-07-17T12:18:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.272286_57.1", 2020-07-17T12:18:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-12:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.12", 2020-07-17T12:18:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.272286467Z", 2020-07-17T12:18:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_kJPn3awfkoAAv2YoILjX1YX", 2020-07-17T12:18:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.274467_58.1", 2020-07-17T12:18:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-12", 2020-07-17T12:18:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.274467332Z", 2020-07-17T12:18:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_WFmqiEkT2ufoxFeioom356B", 2020-07-17T12:18:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987605.278144_59.1", 2020-07-17T12:18:12.445 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.445 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-12", 2020-07-17T12:18:12.445 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:45.278144599Z", 2020-07-17T12:18:12.445 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.446 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.446 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.446 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.446 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.447 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_WFmqiEkT2ufoxFeioom356B", 2020-07-17T12:18:12.447 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.447 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.447 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.448 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.712640_60.1", 2020-07-17T12:18:12.448 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.449 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-13:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.13", 2020-07-17T12:18:12.449 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.712640382Z", 2020-07-17T12:18:12.449 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.449 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.449 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.449 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.450 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.450 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_coVsWkNqvKBUCWs8AacIoNl", 2020-07-17T12:18:12.450 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.451 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.451 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.452 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.717263_61.1", 2020-07-17T12:18:12.452 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.452 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-13:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.13", 2020-07-17T12:18:12.452 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.717263960Z", 2020-07-17T12:18:12.452 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.453 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.453 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.453 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.453 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.453 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_coVsWkNqvKBUCWs8AacIoNl", 2020-07-17T12:18:12.454 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.454 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.454 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.455 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.455 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.719823_62.1", 2020-07-17T12:18:12.456 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.456 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-13", 2020-07-17T12:18:12.456 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.719823082Z", 2020-07-17T12:18:12.456 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.456 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.456 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.457 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.457 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.457 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dWoVeSt6MXyutBS6iBOB_SN", 2020-07-17T12:18:12.457 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.458 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.458 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.459 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.725332_63.1", 2020-07-17T12:18:12.459 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.459 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-13", 2020-07-17T12:18:12.459 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.725332531Z", 2020-07-17T12:18:12.460 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.460 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.460 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.460 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.460 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.461 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dWoVeSt6MXyutBS6iBOB_SN", 2020-07-17T12:18:12.461 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.461 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.461 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.462 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.756675_64.1", 2020-07-17T12:18:12.462 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.463 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-14:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.14", 2020-07-17T12:18:12.463 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.756675726Z", 2020-07-17T12:18:12.463 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.463 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.463 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.464 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.464 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.464 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_BWPfn12CXl2TlU4AWhdAySv", 2020-07-17T12:18:12.464 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.465 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.465 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.465 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.465 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.466 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.761544_65.1", 2020-07-17T12:18:12.466 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.466 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-14:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.14", 2020-07-17T12:18:12.466 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.761544051Z", 2020-07-17T12:18:12.467 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.467 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.467 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.467 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.467 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.468 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_BWPfn12CXl2TlU4AWhdAySv", 2020-07-17T12:18:12.468 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.468 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.469 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.469 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.469 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.470 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.763526_66.1", 2020-07-17T12:18:12.470 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.470 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-14", 2020-07-17T12:18:12.470 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.763526945Z", 2020-07-17T12:18:12.470 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.470 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.471 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.471 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.471 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.471 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_irk8P5j0CkMCoEl6rP2eANQ", 2020-07-17T12:18:12.472 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.472 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.472 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.473 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.767394_67.1", 2020-07-17T12:18:12.473 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.473 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-14", 2020-07-17T12:18:12.474 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.767394556Z", 2020-07-17T12:18:12.474 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.474 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.474 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.474 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.475 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.475 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_irk8P5j0CkMCoEl6rP2eANQ", 2020-07-17T12:18:12.475 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.475 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.476 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.476 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.810791_68.1", 2020-07-17T12:18:12.477 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.477 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-15:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.15", 2020-07-17T12:18:12.477 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.810791998Z", 2020-07-17T12:18:12.477 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.477 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.478 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.478 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.478 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.478 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_aYP5BWVqE2ZQzg8KQUMoKIk", 2020-07-17T12:18:12.479 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.479 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.479 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.479 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.481 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.816286_69.1", 2020-07-17T12:18:12.481 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.481 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-15:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.15", 2020-07-17T12:18:12.481 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.816286566Z", 2020-07-17T12:18:12.482 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.482 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.482 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.482 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.483 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.483 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_aYP5BWVqE2ZQzg8KQUMoKIk", 2020-07-17T12:18:12.483 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.483 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.484 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.485 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.820905_70.1", 2020-07-17T12:18:12.486 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.486 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-15", 2020-07-17T12:18:12.486 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.820905407Z", 2020-07-17T12:18:12.486 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.486 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.487 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.487 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.487 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.487 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_rpYWMBKFNIdTSlnC72SKitx", 2020-07-17T12:18:12.488 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.488 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.488 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.489 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.490 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987618.830894_71.1", 2020-07-17T12:18:12.490 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.490 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-15", 2020-07-17T12:18:12.491 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:58.830894206Z", 2020-07-17T12:18:12.491 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.491 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.492 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.492 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.493 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.493 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_rpYWMBKFNIdTSlnC72SKitx", 2020-07-17T12:18:12.493 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.494 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.494 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.496 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.232376_72.1", 2020-07-17T12:18:12.496 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.496 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:12.497 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.232376434Z", 2020-07-17T12:18:12.497 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.497 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.498 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.498 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.499 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.500 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:12.500 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.501 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.501 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.502 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.503 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.236895_73.1", 2020-07-17T12:18:12.503 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.503 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:12.504 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.236895263Z", 2020-07-17T12:18:12.504 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.504 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.504 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.505 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.505 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.506 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:12.506 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.507 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.507 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.508 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.509 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.238681_74.1", 2020-07-17T12:18:12.509 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.509 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16", 2020-07-17T12:18:12.510 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.238681635Z", 2020-07-17T12:18:12.510 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.510 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.510 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.511 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.511 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.512 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_lEb4NYqynMKhYQrCh3ur9Pg", 2020-07-17T12:18:12.512 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.512 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.512 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.513 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.242366_75.1", 2020-07-17T12:18:12.514 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.514 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16", 2020-07-17T12:18:12.514 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.242366508Z", 2020-07-17T12:18:12.514 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.514 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.515 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.515 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.515 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.516 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_lEb4NYqynMKhYQrCh3ur9Pg", 2020-07-17T12:18:12.516 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.516 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.516 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.517 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.269805_76.1", 2020-07-17T12:18:12.518 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.518 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:12.518 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.269805039Z", 2020-07-17T12:18:12.518 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.518 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.519 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.519 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.519 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.519 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:12.519 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.520 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.520 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.521 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.273253_77.1", 2020-07-17T12:18:12.521 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.521 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:12.522 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.273253052Z", 2020-07-17T12:18:12.522 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.522 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.522 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.522 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.523 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.523 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:12.523 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.524 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.524 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.525 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.274824_78.1", 2020-07-17T12:18:12.525 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.525 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17", 2020-07-17T12:18:12.525 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.274824384Z", 2020-07-17T12:18:12.525 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.526 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.526 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.526 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.526 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.527 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_kC34LrBQXhenfmJdKuMdgnY", 2020-07-17T12:18:12.527 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.527 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.527 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.528 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.528 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.277982_79.1", 2020-07-17T12:18:12.529 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.529 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17", 2020-07-17T12:18:12.529 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.277982524Z", 2020-07-17T12:18:12.529 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.529 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.530 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.530 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.530 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.530 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_kC34LrBQXhenfmJdKuMdgnY", 2020-07-17T12:18:12.530 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.531 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.531 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.532 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.321344_80.1", 2020-07-17T12:18:12.532 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.533 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:12.533 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.321344326Z", 2020-07-17T12:18:12.533 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.533 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.533 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.533 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.534 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.534 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:12.534 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.535 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.535 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.535 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.536 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.325975_81.1", 2020-07-17T12:18:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.325975985Z", 2020-07-17T12:18:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.537 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:12.538 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.327813_82.1", 2020-07-17T12:18:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18", 2020-07-17T12:18:12.540 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.327813080Z", 2020-07-17T12:18:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.541 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_0c-EMQmUZNyncsOLFYYvBXZ", 2020-07-17T12:18:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.331302_83.1", 2020-07-17T12:18:12.543 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.544 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18", 2020-07-17T12:18:12.544 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.331302239Z", 2020-07-17T12:18:12.544 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.544 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.545 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.545 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.545 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.545 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_0c-EMQmUZNyncsOLFYYvBXZ", 2020-07-17T12:18:12.545 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.546 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.546 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.547 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.349661_84.1", 2020-07-17T12:18:12.547 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.547 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:12.547 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.349661555Z", 2020-07-17T12:18:12.548 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.548 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.548 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:12.548 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.549 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.549 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:12.549 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.549 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.549 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.550 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.354642_85.1", 2020-07-17T12:18:12.551 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.551 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:12.551 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.354642695Z", 2020-07-17T12:18:12.551 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.551 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.551 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:12.552 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.552 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.552 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:12.552 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.553 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.553 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.554 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.358809_86.1", 2020-07-17T12:18:12.554 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.554 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:12.554 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.358809552Z", 2020-07-17T12:18:12.555 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.555 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.555 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:12.555 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.555 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.556 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:12.556 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.556 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.556 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.557 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.364350_87.1", 2020-07-17T12:18:12.557 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.558 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:12.558 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.364350896Z", 2020-07-17T12:18:12.558 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.558 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.558 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:12.558 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.559 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.559 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:12.559 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.560 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.560 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.561 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.374842_88.1", 2020-07-17T12:18:12.561 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.561 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:12.562 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.374842199Z", 2020-07-17T12:18:12.562 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.562 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.562 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:12.562 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.563 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.563 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:12.563 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.563 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.564 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.565 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987653.381436_89.1", 2020-07-17T12:18:12.565 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.566 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:12.566 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:07:33.381436296Z", 2020-07-17T12:18:12.566 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.566 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.567 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:12.567 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.568 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.568 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:12.568 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.569 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.569 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.571 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.580189_90.1", 2020-07-17T12:18:12.571 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.572 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:12.572 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.580189016Z", 2020-07-17T12:18:12.573 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.573 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.574 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.574 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.576 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.576 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.578 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.579 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.579 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.580 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.583928_91.1", 2020-07-17T12:18:12.580 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.581 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:12.581 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.583928438Z", 2020-07-17T12:18:12.581 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.581 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.581 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.582 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.582 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.582 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:12.583 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.583 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.583 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.584 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.585595_92.1", 2020-07-17T12:18:12.584 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.585 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19", 2020-07-17T12:18:12.585 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.585595449Z", 2020-07-17T12:18:12.585 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.585 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.585 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.586 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.586 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.586 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Y0anvog_R4e6DQW9utotNY_", 2020-07-17T12:18:12.587 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.587 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.587 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.587 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.588 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.589056_93.1", 2020-07-17T12:18:12.588 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.589 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19", 2020-07-17T12:18:12.589 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.589056210Z", 2020-07-17T12:18:12.589 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.589 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.589 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.590 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.590 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.590 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Y0anvog_R4e6DQW9utotNY_", 2020-07-17T12:18:12.590 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.591 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.591 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.592 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.596888_94.1", 2020-07-17T12:18:12.592 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.592 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:12.593 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.596888308Z", 2020-07-17T12:18:12.593 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.593 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.593 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:12.593 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.594 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.594 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:12.594 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.594 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.595 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.595 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987843.602473_95.1", 2020-07-17T12:18:12.596 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.596 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:12.596 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:43.602473621Z", 2020-07-17T12:18:12.596 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.596 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.597 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:12.597 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.597 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.597 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:12.597 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.598 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.598 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.598 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.598 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.599 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987998.537142_96.1", 2020-07-17T12:18:12.599 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.600 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:12.600 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:18.537142107Z", 2020-07-17T12:18:12.601 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.601 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.601 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.601 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.602 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.602 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.603 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.603 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.604 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.605 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987998.541926_97.1", 2020-07-17T12:18:12.605 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.606 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:12.606 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:18.541926282Z", 2020-07-17T12:18:12.606 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.607 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.607 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.607 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.608 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.608 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.608 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.609 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.609 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.618 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987998.544300_98.1", 2020-07-17T12:18:12.619 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.619 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20", 2020-07-17T12:18:12.619 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:18.544300322Z", 2020-07-17T12:18:12.620 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.620 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.620 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.621 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.621 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.621 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_qoKh-ps-xZklh5sIuAPFkjZ", 2020-07-17T12:18:12.622 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.622 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.623 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.624 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987998.548993_99.1", 2020-07-17T12:18:12.625 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.625 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20", 2020-07-17T12:18:12.625 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:18.548993097Z", 2020-07-17T12:18:12.626 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.626 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.626 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.627 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.627 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.628 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_qoKh-ps-xZklh5sIuAPFkjZ", 2020-07-17T12:18:12.628 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.629 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.629 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.631 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988004.926158_100.1", 2020-07-17T12:18:12.631 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.632 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:12.632 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:24.926158716Z", 2020-07-17T12:18:12.633 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.633 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.633 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.634 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.634 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.635 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.635 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.638 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988004.936521_101.1", 2020-07-17T12:18:12.638 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.639 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:12.639 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:13:24.936521959Z", 2020-07-17T12:18:12.639 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.640 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.640 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.640 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.641 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.641 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.642 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.642 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.643 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.643 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.643 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.644 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988065.046888_102.1", 2020-07-17T12:18:12.645 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.645 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:12.645 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:14:25.046888654Z", 2020-07-17T12:18:12.646 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.646 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.646 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.647 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.647 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.648 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.648 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:12.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.649 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.649 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.650 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.650 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.651 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988065.056306_103.1", 2020-07-17T12:18:12.651 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.652 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:12.652 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:14:25.056306412Z", 2020-07-17T12:18:12.652 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.652 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.652 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.653 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.653 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.654 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:12.654 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:12.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.654 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.654 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.656 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.108561_114.1", 2020-07-17T12:18:12.656 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.656 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:12.657 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.108561553Z", 2020-07-17T12:18:12.657 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.657 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.657 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.657 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.658 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.658 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:12.659 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.659 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.659 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.659 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.660 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.112561_115.1", 2020-07-17T12:18:12.660 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.661 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:12.661 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.112561600Z", 2020-07-17T12:18:12.661 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.661 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.661 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.662 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.662 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.662 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:12.662 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.663 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.663 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.664 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.114341_116.1", 2020-07-17T12:18:12.664 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.664 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21", 2020-07-17T12:18:12.664 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.114341527Z", 2020-07-17T12:18:12.665 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.665 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.665 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.665 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.666 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.666 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_7IHshvdvzhTN_iaS63UlhhP", 2020-07-17T12:18:12.666 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.666 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.667 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.668 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.118095_117.1", 2020-07-17T12:18:12.668 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.668 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21", 2020-07-17T12:18:12.668 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.118095935Z", 2020-07-17T12:18:12.668 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.668 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.669 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.669 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.669 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.669 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_7IHshvdvzhTN_iaS63UlhhP", 2020-07-17T12:18:12.670 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.670 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.670 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.670 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.671 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.144536_119.1", 2020-07-17T12:18:12.671 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.671 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:12.672 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.144536481Z", 2020-07-17T12:18:12.672 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.672 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.672 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.672 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.673 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.673 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:12.673 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.674 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.674 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.674 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.674 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.675 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.148697_120.1", 2020-07-17T12:18:12.675 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.676 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:12.676 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.148697190Z", 2020-07-17T12:18:12.676 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.676 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.677 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.677 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.678 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.678 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:12.678 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.679 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.679 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.680 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.150760_121.1", 2020-07-17T12:18:12.680 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.680 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22", 2020-07-17T12:18:12.680 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.150760146Z", 2020-07-17T12:18:12.681 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.681 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.681 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.681 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.682 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.682 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gsPtnlqFRKAZ04Ktvzs95UN", 2020-07-17T12:18:12.682 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.683 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.683 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.684 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.155597_122.1", 2020-07-17T12:18:12.684 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.684 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22", 2020-07-17T12:18:12.684 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.155597648Z", 2020-07-17T12:18:12.684 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.685 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.685 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.685 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.685 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gsPtnlqFRKAZ04Ktvzs95UN", 2020-07-17T12:18:12.686 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.686 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.686 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.687 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.195741_123.1", 2020-07-17T12:18:12.687 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.688 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:12.688 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.195741697Z", 2020-07-17T12:18:12.688 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.688 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.688 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.689 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.689 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.689 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:12.689 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.690 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.690 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.691 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.200019_124.1", 2020-07-17T12:18:12.691 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.691 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:12.691 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.200019763Z", 2020-07-17T12:18:12.691 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.692 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.692 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.692 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.693 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:12.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.693 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.693 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.694 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.694 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.201967_125.1", 2020-07-17T12:18:12.694 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.695 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23", 2020-07-17T12:18:12.695 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.201967975Z", 2020-07-17T12:18:12.695 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.695 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.695 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.696 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.696 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_4EWRUD1pbHkY-FiDIwWfNaQ", 2020-07-17T12:18:12.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.697 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.697 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.698 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.205751_126.1", 2020-07-17T12:18:12.698 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.698 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23", 2020-07-17T12:18:12.698 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.205751555Z", 2020-07-17T12:18:12.699 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.699 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.699 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.699 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.701 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_4EWRUD1pbHkY-FiDIwWfNaQ", 2020-07-17T12:18:12.701 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.701 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.701 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.701 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.702 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.226616_127.1", 2020-07-17T12:18:12.702 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.703 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:12.703 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.226616279Z", 2020-07-17T12:18:12.703 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.703 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.703 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:12.704 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.704 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.704 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:12.704 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.705 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.705 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.706 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.231665_128.1", 2020-07-17T12:18:12.706 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.706 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:12.706 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.231665701Z", 2020-07-17T12:18:12.707 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.707 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:12.707 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.707 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.708 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:12.708 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.708 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.708 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.709 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.236032_130.1", 2020-07-17T12:18:12.710 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.710 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:12.710 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.236032642Z", 2020-07-17T12:18:12.710 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.711 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.711 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:12.711 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.712 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.712 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:12.713 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.713 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.714 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.714 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.714 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.715 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.240206_131.1", 2020-07-17T12:18:12.716 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.716 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:12.716 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.240206565Z", 2020-07-17T12:18:12.717 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.717 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.717 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:12.718 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.719 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:12.720 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.720 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.721 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.722 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.251296_132.1", 2020-07-17T12:18:12.723 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.723 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:12.723 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.251296772Z", 2020-07-17T12:18:12.724 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.724 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.724 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:12.725 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.725 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.726 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:12.726 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.727 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.727 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.729 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988109.262832_133.1", 2020-07-17T12:18:12.730 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.730 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:12.730 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:09.262832933Z", 2020-07-17T12:18:12.731 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.731 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.731 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:12.732 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.732 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.733 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:12.733 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.734 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.734 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.734 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.736 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.216266_162.1", 2020-07-17T12:18:12.736 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.736 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:12.736 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.216266874Z", 2020-07-17T12:18:12.737 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.737 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.737 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.738 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.738 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.739 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.739 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.740 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.740 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.742 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.220314_163.1", 2020-07-17T12:18:12.742 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.742 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:12.743 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.220314597Z", 2020-07-17T12:18:12.743 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.743 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.744 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.748 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.222492_164.1", 2020-07-17T12:18:12.748 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.749 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24", 2020-07-17T12:18:12.749 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.222492473Z", 2020-07-17T12:18:12.749 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.750 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.750 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.750 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.751 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.751 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1aTUtnO5v7_BzTHJ7dcWax9", 2020-07-17T12:18:12.752 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.752 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.753 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.754 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.226266_165.1", 2020-07-17T12:18:12.755 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.755 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24", 2020-07-17T12:18:12.755 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.226266565Z", 2020-07-17T12:18:12.756 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.756 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.756 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.756 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.757 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.757 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1aTUtnO5v7_BzTHJ7dcWax9", 2020-07-17T12:18:12.758 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.759 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.759 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.761 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.254586_166.1", 2020-07-17T12:18:12.761 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.761 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:12.762 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.254586031Z", 2020-07-17T12:18:12.762 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.762 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.762 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.763 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.763 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.764 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.764 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.765 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.765 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.766 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.767 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.259320_167.1", 2020-07-17T12:18:12.767 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.767 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:12.768 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.259320320Z", 2020-07-17T12:18:12.768 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.768 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.768 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.769 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.770 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.770 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.771 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.771 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.772 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.261985_168.1", 2020-07-17T12:18:12.772 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.772 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25", 2020-07-17T12:18:12.772 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.261985157Z", 2020-07-17T12:18:12.773 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.773 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.773 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.773 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.774 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.774 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_CJpPPaq5s3R_DLWK7_Dw6N9", 2020-07-17T12:18:12.774 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.774 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.775 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.776 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.268842_169.1", 2020-07-17T12:18:12.776 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.776 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25", 2020-07-17T12:18:12.777 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.268842944Z", 2020-07-17T12:18:12.777 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.777 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.777 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.777 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.778 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.778 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_CJpPPaq5s3R_DLWK7_Dw6N9", 2020-07-17T12:18:12.778 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.779 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.779 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.780 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.781 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.310784_170.1", 2020-07-17T12:18:12.781 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.781 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:12.781 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.310784639Z", 2020-07-17T12:18:12.782 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.782 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.782 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.783 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.783 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.784 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.785 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.785 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.787 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.319433_171.1", 2020-07-17T12:18:12.787 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.788 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:12.788 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.319433996Z", 2020-07-17T12:18:12.788 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.789 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.789 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.789 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.790 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.790 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.791 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.791 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.791 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.793 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.322090_172.1", 2020-07-17T12:18:12.793 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.794 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26", 2020-07-17T12:18:12.794 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.322090964Z", 2020-07-17T12:18:12.794 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.795 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.795 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.795 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.796 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.796 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_OI4YC_rV8-IvJU_ONTZMZ22", 2020-07-17T12:18:12.797 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.797 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.798 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.800 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.326100_173.1", 2020-07-17T12:18:12.800 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.800 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26", 2020-07-17T12:18:12.800 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.326100789Z", 2020-07-17T12:18:12.801 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.801 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.801 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.802 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.802 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.803 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_OI4YC_rV8-IvJU_ONTZMZ22", 2020-07-17T12:18:12.803 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.804 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.804 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.805 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.345795_174.1", 2020-07-17T12:18:12.806 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.806 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:12.806 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.345795397Z", 2020-07-17T12:18:12.807 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.807 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.807 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.808 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.808 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.809 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.809 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.810 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.810 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.811 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.351197_175.1", 2020-07-17T12:18:12.811 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.811 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:12.812 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.351197071Z", 2020-07-17T12:18:12.812 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.812 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.812 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.812 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.813 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.813 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.813 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.813 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.814 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.815 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.354716_176.1", 2020-07-17T12:18:12.815 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.822 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:12.822 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.354716106Z", 2020-07-17T12:18:12.822 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.822 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.822 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.823 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.823 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.823 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.823 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:12.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.824 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.824 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.825 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.360435_177.1", 2020-07-17T12:18:12.825 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.826 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:12.826 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.360435248Z", 2020-07-17T12:18:12.826 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.826 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.826 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.826 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.827 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.827 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:12.827 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:12.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.828 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.828 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.829 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.365795_178.1", 2020-07-17T12:18:12.829 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.829 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:12.829 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.365795374Z", 2020-07-17T12:18:12.829 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.830 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.830 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.830 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.830 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.830 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.831 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.831 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.831 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.832 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.372555_179.1", 2020-07-17T12:18:12.832 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.832 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:12.833 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.372555105Z", 2020-07-17T12:18:12.833 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.833 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.833 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.833 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.834 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.834 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.834 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.834 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.835 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.835 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.835 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.836 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.385196_180.1", 2020-07-17T12:18:12.836 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.836 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:12.836 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.385196882Z", 2020-07-17T12:18:12.836 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.836 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.837 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.837 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.837 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.837 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.838 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:12.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.838 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.838 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.839 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.393417_181.1", 2020-07-17T12:18:12.839 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.839 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:12.839 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.393417880Z", 2020-07-17T12:18:12.840 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.840 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.840 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.841 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.841 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:12.841 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:12.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.841 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.841 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.842 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.405697_182.1", 2020-07-17T12:18:12.843 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.843 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:12.843 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.405697153Z", 2020-07-17T12:18:12.843 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.843 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.843 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.844 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.844 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.844 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.844 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.845 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.845 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.846 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.415076_183.1", 2020-07-17T12:18:12.846 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.847 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:12.847 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.415076707Z", 2020-07-17T12:18:12.847 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.847 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.848 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.848 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.848 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.848 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.849 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.849 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.849 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.850 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.426554_184.1", 2020-07-17T12:18:12.850 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.851 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:12.851 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.426554025Z", 2020-07-17T12:18:12.851 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.851 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.851 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.852 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.852 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.852 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:12.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.853 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.853 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.854 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988126.433699_185.1", 2020-07-17T12:18:12.854 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.854 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:12.855 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:26.433699189Z", 2020-07-17T12:18:12.855 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.855 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.855 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.855 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.856 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:12.856 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 3 2020-07-17T12:18:12.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.856 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.857 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.858 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.192609_186.1", 2020-07-17T12:18:12.858 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.858 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:12.858 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.192609023Z", 2020-07-17T12:18:12.858 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.858 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.859 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.859 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.859 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.859 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:12.860 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.860 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.860 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.861 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.197302_187.1", 2020-07-17T12:18:12.861 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.861 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:12.862 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.197302293Z", 2020-07-17T12:18:12.862 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.862 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.862 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.862 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.863 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.863 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:12.863 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.863 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.864 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.864 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.200146_188.1", 2020-07-17T12:18:12.865 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.865 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27", 2020-07-17T12:18:12.865 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.200146111Z", 2020-07-17T12:18:12.865 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.865 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.866 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.866 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.866 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.866 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_EAM3oic4BNiOLUongn3X0x8", 2020-07-17T12:18:12.867 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.867 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.867 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.868 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.204265_189.1", 2020-07-17T12:18:12.868 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.868 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27", 2020-07-17T12:18:12.868 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.204265093Z", 2020-07-17T12:18:12.869 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.869 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.869 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.869 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.870 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.870 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_EAM3oic4BNiOLUongn3X0x8", 2020-07-17T12:18:12.870 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.870 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.870 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.871 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.232604_190.1", 2020-07-17T12:18:12.872 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.872 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:12.872 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.232604283Z", 2020-07-17T12:18:12.872 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.872 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.872 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.873 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.873 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.873 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:12.873 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.874 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.874 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.875 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.236653_191.1", 2020-07-17T12:18:12.875 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.875 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:12.876 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.236653061Z", 2020-07-17T12:18:12.876 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.876 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:12.877 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.238308_192.1", 2020-07-17T12:18:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28", 2020-07-17T12:18:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.238308219Z", 2020-07-17T12:18:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_8tICQSSPhT4sUiRts2CGtXF", 2020-07-17T12:18:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.242121_193.1", 2020-07-17T12:18:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28", 2020-07-17T12:18:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.242121826Z", 2020-07-17T12:18:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_8tICQSSPhT4sUiRts2CGtXF", 2020-07-17T12:18:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.291427_194.1", 2020-07-17T12:18:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.291427038Z", 2020-07-17T12:18:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.889 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.296400_195.1", 2020-07-17T12:18:12.889 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.890 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:12.890 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.296400519Z", 2020-07-17T12:18:12.890 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.890 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.890 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.891 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.891 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.891 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:12.891 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.892 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.892 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.893 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.299100_196.1", 2020-07-17T12:18:12.893 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.893 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29", 2020-07-17T12:18:12.893 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.299100799Z", 2020-07-17T12:18:12.893 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.894 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.894 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.894 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.894 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.894 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oUHOV9TdQ-W-rj70_Cou9X5", 2020-07-17T12:18:12.895 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.895 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.895 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.896 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.304751_197.1", 2020-07-17T12:18:12.896 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.896 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29", 2020-07-17T12:18:12.897 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.304751236Z", 2020-07-17T12:18:12.897 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.897 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.897 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.897 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.898 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.898 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oUHOV9TdQ-W-rj70_Cou9X5", 2020-07-17T12:18:12.898 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.899 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.899 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.900 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.329636_198.1", 2020-07-17T12:18:12.900 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.900 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:12.900 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.329636096Z", 2020-07-17T12:18:12.900 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.901 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:12.901 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.901 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.902 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:12.902 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.902 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.902 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.902 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.903 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.334331_199.1", 2020-07-17T12:18:12.903 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.904 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:12.904 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.334331520Z", 2020-07-17T12:18:12.904 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.904 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.904 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:12.904 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.905 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.906 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:12.906 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.906 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.906 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.907 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.388479_200.1", 2020-07-17T12:18:12.908 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.908 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:12.908 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.388479756Z", 2020-07-17T12:18:12.908 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.908 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.908 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:12.909 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.909 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.909 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:12.909 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.910 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.910 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.911 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.398272_201.1", 2020-07-17T12:18:12.911 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.911 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:12.911 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.398272953Z", 2020-07-17T12:18:12.912 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.912 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.912 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:12.912 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.915 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.915 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:12.916 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.916 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.917 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.919 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.463878_202.1", 2020-07-17T12:18:12.919 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.919 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:12.920 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.463878801Z", 2020-07-17T12:18:12.920 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.920 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.921 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:12.921 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.922 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:12.923 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.924 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.924 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.926 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.474476_203.1", 2020-07-17T12:18:12.927 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.927 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:12.927 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.474476699Z", 2020-07-17T12:18:12.928 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.928 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.928 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:12.929 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.929 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.930 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:12.930 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.931 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.931 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.932 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.932 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.933 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.543823_204.1", 2020-07-17T12:18:12.933 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.933 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:12.934 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.543823468Z", 2020-07-17T12:18:12.934 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.934 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.935 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.935 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.936 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.936 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:12.936 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.936 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.937 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.938 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.548616_205.1", 2020-07-17T12:18:12.938 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.938 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:12.938 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.548616662Z", 2020-07-17T12:18:12.938 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.939 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.939 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.939 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.939 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.939 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:12.940 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.940 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.940 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.941 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.941 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.550563_206.1", 2020-07-17T12:18:12.941 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.941 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30", 2020-07-17T12:18:12.942 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.550563904Z", 2020-07-17T12:18:12.942 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.942 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.942 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.942 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.943 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.944 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_lJ2S3jRGkcD2jZEcb0bU-NY", 2020-07-17T12:18:12.944 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.945 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.945 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.945 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.945 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.946 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.555228_207.1", 2020-07-17T12:18:12.946 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.946 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30", 2020-07-17T12:18:12.946 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.555228975Z", 2020-07-17T12:18:12.946 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.947 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.947 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.947 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.947 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.948 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_lJ2S3jRGkcD2jZEcb0bU-NY", 2020-07-17T12:18:12.954 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.954 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.954 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.954 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.955 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.579434_208.1", 2020-07-17T12:18:12.955 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.956 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:12.956 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.579434456Z", 2020-07-17T12:18:12.956 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.956 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.956 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.957 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.957 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.957 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:12.957 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.958 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.958 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.958 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.958 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.959 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.583820_209.1", 2020-07-17T12:18:12.959 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.959 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:12.959 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.583820075Z", 2020-07-17T12:18:12.960 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.960 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.960 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.960 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.961 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.961 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:12.961 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.961 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.962 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.963 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.585899_210.1", 2020-07-17T12:18:12.963 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.963 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31", 2020-07-17T12:18:12.963 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.585899826Z", 2020-07-17T12:18:12.963 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.964 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.964 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.964 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.964 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.965 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_ln1eOUq8lDkVI4IpQvkNAi4", 2020-07-17T12:18:12.965 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.965 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.965 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.966 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.589417_211.1", 2020-07-17T12:18:12.967 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.967 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31", 2020-07-17T12:18:12.967 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.589417398Z", 2020-07-17T12:18:12.967 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.967 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.967 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.968 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.968 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.968 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_ln1eOUq8lDkVI4IpQvkNAi4", 2020-07-17T12:18:12.968 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.969 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.969 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.970 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.638248_212.1", 2020-07-17T12:18:12.970 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.970 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:12.970 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.638248979Z", 2020-07-17T12:18:12.970 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.971 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.971 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.971 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.971 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.972 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:12.972 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.972 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.972 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.973 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.642956_213.1", 2020-07-17T12:18:12.973 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.974 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:12.974 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.642956157Z", 2020-07-17T12:18:12.974 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.974 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.974 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.974 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.975 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.975 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:12.975 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.976 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.976 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.977 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.644834_214.1", 2020-07-17T12:18:12.977 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.978 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32", 2020-07-17T12:18:12.978 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.644834814Z", 2020-07-17T12:18:12.978 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.978 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.978 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.978 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.979 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_CTGkGtU8NTFdGu11wP4HBlE", 2020-07-17T12:18:12.979 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.979 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.980 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.981 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.649548_215.1", 2020-07-17T12:18:12.981 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:12.981 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32", 2020-07-17T12:18:12.981 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.649548275Z", 2020-07-17T12:18:12.981 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.981 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.982 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:12.982 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:12.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.982 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.982 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_CTGkGtU8NTFdGu11wP4HBlE", 2020-07-17T12:18:12.983 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.983 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.983 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.983 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.983 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.984 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.691484_216.1", 2020-07-17T12:18:12.984 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.984 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:12.985 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.691484060Z", 2020-07-17T12:18:12.985 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.985 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.985 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:12.985 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.986 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.986 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:12.986 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.986 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.987 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.988 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.695812_217.1", 2020-07-17T12:18:12.988 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.989 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:12.989 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.695812573Z", 2020-07-17T12:18:12.990 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.990 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.990 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:12.990 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.991 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.991 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:12.991 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.991 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.752305_218.1", 2020-07-17T12:18:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.752305251Z", 2020-07-17T12:18:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.759930_219.1", 2020-07-17T12:18:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.759930328Z", 2020-07-17T12:18:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.819822_220.1", 2020-07-17T12:18:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.819822109Z", 2020-07-17T12:18:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.002 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.002 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:13.002 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.003 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.003 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.004 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.829887_221.1", 2020-07-17T12:18:13.004 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.004 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:13.004 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.829887197Z", 2020-07-17T12:18:13.004 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.005 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.005 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:13.005 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.006 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.006 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:13.006 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.006 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.007 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.008 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.897711_222.1", 2020-07-17T12:18:13.008 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.009 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:13.009 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.897711412Z", 2020-07-17T12:18:13.009 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.009 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.010 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.010 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.010 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.010 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:13.011 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.011 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.011 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.011 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.011 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.012 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.901139_223.1", 2020-07-17T12:18:13.012 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.012 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:13.013 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.901139991Z", 2020-07-17T12:18:13.013 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.013 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.013 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.013 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.014 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.014 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:13.015 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.015 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.015 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.017 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.902679_224.1", 2020-07-17T12:18:13.017 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.018 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33", 2020-07-17T12:18:13.018 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.902679722Z", 2020-07-17T12:18:13.018 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.019 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.019 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.019 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.020 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.020 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_UxWO-19hsf0zfhgE2d7C0Bs", 2020-07-17T12:18:13.020 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.021 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.021 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.023 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.906270_225.1", 2020-07-17T12:18:13.023 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.023 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33", 2020-07-17T12:18:13.023 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.906270781Z", 2020-07-17T12:18:13.024 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.024 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.024 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.025 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.025 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.026 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_UxWO-19hsf0zfhgE2d7C0Bs", 2020-07-17T12:18:13.026 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.027 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.027 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.028 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.932251_226.1", 2020-07-17T12:18:13.028 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.029 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:13.029 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.932251093Z", 2020-07-17T12:18:13.029 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.030 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.030 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.030 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.031 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.031 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:13.031 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.032 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.032 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.034 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.937747_227.1", 2020-07-17T12:18:13.034 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.035 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:13.035 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.937747244Z", 2020-07-17T12:18:13.035 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.036 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.036 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.036 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:13.037 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.038 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.939489_228.1", 2020-07-17T12:18:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34", 2020-07-17T12:18:13.040 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.939489887Z", 2020-07-17T12:18:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.041 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.042 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oKhT8xShmiPT7_3DwLLaiZf", 2020-07-17T12:18:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.942587_229.1", 2020-07-17T12:18:13.045 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34", 2020-07-17T12:18:13.046 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.942587543Z", 2020-07-17T12:18:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.047 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_oKhT8xShmiPT7_3DwLLaiZf", 2020-07-17T12:18:13.048 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.977552_230.1", 2020-07-17T12:18:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:13.051 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.977552760Z", 2020-07-17T12:18:13.052 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.052 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.052 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.053 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:13.054 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.054 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.056 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.982605_231.1", 2020-07-17T12:18:13.056 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.982605704Z", 2020-07-17T12:18:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.057 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.058 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.059 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:13.059 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.061 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.985248_232.1", 2020-07-17T12:18:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35", 2020-07-17T12:18:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.985248699Z", 2020-07-17T12:18:13.062 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_vxH0cpq2NUCS2-F5AwiLlGy", 2020-07-17T12:18:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988148.989554_233.1", 2020-07-17T12:18:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.066 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35", 2020-07-17T12:18:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:48.989554870Z", 2020-07-17T12:18:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.067 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_vxH0cpq2NUCS2-F5AwiLlGy", 2020-07-17T12:18:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988149.009800_234.1", 2020-07-17T12:18:13.070 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:49.009800265Z", 2020-07-17T12:18:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.071 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:13.073 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988149.014525_235.1", 2020-07-17T12:18:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.075 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:49.014525305Z", 2020-07-17T12:18:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.076 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.077 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.078 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988150.064957_236.1", 2020-07-17T12:18:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:50.064957041Z", 2020-07-17T12:18:13.080 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.081 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.082 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988150.071666_237.1", 2020-07-17T12:18:13.083 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:50.071666653Z", 2020-07-17T12:18:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.084 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:13.085 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.128899_238.1", 2020-07-17T12:18:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:13.087 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.128899145Z", 2020-07-17T12:18:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.089 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.136044_239.1", 2020-07-17T12:18:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:13.091 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.136044802Z", 2020-07-17T12:18:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.093 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.094 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.223773_240.1", 2020-07-17T12:18:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-36:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.36", 2020-07-17T12:18:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.223773094Z", 2020-07-17T12:18:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.095 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jdX6lkwwXVktMAEuRsLdL9f", 2020-07-17T12:18:13.096 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.227459_241.1", 2020-07-17T12:18:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-36:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.36", 2020-07-17T12:18:13.098 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.227459296Z", 2020-07-17T12:18:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jdX6lkwwXVktMAEuRsLdL9f", 2020-07-17T12:18:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.229174_242.1", 2020-07-17T12:18:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-36", 2020-07-17T12:18:13.102 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.229174371Z", 2020-07-17T12:18:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_qgrqKYBWaNJyRRn3Vjx0wVN", 2020-07-17T12:18:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.104 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.232729_243.1", 2020-07-17T12:18:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-36", 2020-07-17T12:18:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.232729435Z", 2020-07-17T12:18:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.106 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.107 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_qgrqKYBWaNJyRRn3Vjx0wVN", 2020-07-17T12:18:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.269955_244.1", 2020-07-17T12:18:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.109 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-37:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.37", 2020-07-17T12:18:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.269955795Z", 2020-07-17T12:18:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_c-tGp8YfI6B1yBpFUhIj44i", 2020-07-17T12:18:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.111 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.112 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.275803_245.1", 2020-07-17T12:18:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-37:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.37", 2020-07-17T12:18:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.275803241Z", 2020-07-17T12:18:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.113 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_c-tGp8YfI6B1yBpFUhIj44i", 2020-07-17T12:18:13.114 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.278648_246.1", 2020-07-17T12:18:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-37", 2020-07-17T12:18:13.116 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.278648356Z", 2020-07-17T12:18:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_9F_FGhwZ3EnXzZglCBoTwNA", 2020-07-17T12:18:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.118 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.285189_247.1", 2020-07-17T12:18:13.119 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-37", 2020-07-17T12:18:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.285189703Z", 2020-07-17T12:18:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.120 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_9F_FGhwZ3EnXzZglCBoTwNA", 2020-07-17T12:18:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.334180_248.1", 2020-07-17T12:18:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-38:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.38", 2020-07-17T12:18:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.334180801Z", 2020-07-17T12:18:13.123 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.124 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_hVnb1ngvk17nkYrOuLaCqwM", 2020-07-17T12:18:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.125 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.126 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.337583_249.1", 2020-07-17T12:18:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-38:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.38", 2020-07-17T12:18:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.337583158Z", 2020-07-17T12:18:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.127 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.128 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_hVnb1ngvk17nkYrOuLaCqwM", 2020-07-17T12:18:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.339086_250.1", 2020-07-17T12:18:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.130 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-38", 2020-07-17T12:18:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.339086250Z", 2020-07-17T12:18:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.131 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_3EGy4bfasMLEwsmYrKxh2v5", 2020-07-17T12:18:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988151.342294_251.1", 2020-07-17T12:18:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.134 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-38", 2020-07-17T12:18:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:51.342294371Z", 2020-07-17T12:18:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.135 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_3EGy4bfasMLEwsmYrKxh2v5", 2020-07-17T12:18:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.136 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.949522_252.1", 2020-07-17T12:18:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-39:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.39", 2020-07-17T12:18:13.138 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.949522585Z", 2020-07-17T12:18:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.139 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gsVvIhPoNrpxq4YxIZghI20", 2020-07-17T12:18:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.140 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.956039_253.1", 2020-07-17T12:18:13.141 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-39:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.39", 2020-07-17T12:18:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.956039623Z", 2020-07-17T12:18:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.142 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.143 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_gsVvIhPoNrpxq4YxIZghI20", 2020-07-17T12:18:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.961169_254.1", 2020-07-17T12:18:13.145 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-39", 2020-07-17T12:18:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.961169072Z", 2020-07-17T12:18:13.146 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.147 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.148 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jHFRYvjwFaeKft8jqkFw9f6", 2020-07-17T12:18:13.148 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.148 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.149 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.965532_255.1", 2020-07-17T12:18:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-39", 2020-07-17T12:18:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.965532718Z", 2020-07-17T12:18:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.151 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jHFRYvjwFaeKft8jqkFw9f6", 2020-07-17T12:18:13.153 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.154 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988152.996911_256.1", 2020-07-17T12:18:13.156 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-40:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.40", 2020-07-17T12:18:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.996911090Z", 2020-07-17T12:18:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.157 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zd9daTaLaaQPrwIpjVSY9v4", 2020-07-17T12:18:13.159 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.162 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.001607_257.1", 2020-07-17T12:18:13.162 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.162 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-40:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.40", 2020-07-17T12:18:13.163 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.001607160Z", 2020-07-17T12:18:13.163 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.163 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.164 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.164 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.164 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.165 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zd9daTaLaaQPrwIpjVSY9v4", 2020-07-17T12:18:13.165 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.166 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.166 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.167 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.004358_258.1", 2020-07-17T12:18:13.168 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.168 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-40", 2020-07-17T12:18:13.168 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.004358398Z", 2020-07-17T12:18:13.168 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.168 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Apkycxzmtorr0IIi3gzU-Qm", 2020-07-17T12:18:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.170 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.170 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.170 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.170 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.171 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.009492_259.1", 2020-07-17T12:18:13.171 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.171 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-40", 2020-07-17T12:18:13.171 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.009492644Z", 2020-07-17T12:18:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.173 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.173 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Apkycxzmtorr0IIi3gzU-Qm", 2020-07-17T12:18:13.173 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.173 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.173 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.174 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.048556_260.1", 2020-07-17T12:18:13.175 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.175 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-41:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.41", 2020-07-17T12:18:13.175 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.048556572Z", 2020-07-17T12:18:13.175 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.175 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.175 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.176 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.176 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.176 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NntBaMu1tW2iuZqebtOm9OP", 2020-07-17T12:18:13.176 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.177 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.177 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.178 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.053032_261.1", 2020-07-17T12:18:13.178 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.178 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-41:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.41", 2020-07-17T12:18:13.179 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.053032571Z", 2020-07-17T12:18:13.179 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.179 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.179 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.179 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.180 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.180 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_NntBaMu1tW2iuZqebtOm9OP", 2020-07-17T12:18:13.180 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.180 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.181 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.182 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.055326_262.1", 2020-07-17T12:18:13.182 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.182 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-41", 2020-07-17T12:18:13.182 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.055326711Z", 2020-07-17T12:18:13.182 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.182 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.183 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.183 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.183 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.183 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_W8htjS8YlHsEASKVqW9GGew", 2020-07-17T12:18:13.184 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.184 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.184 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.184 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.184 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.185 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988153.060681_263.1", 2020-07-17T12:18:13.185 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.185 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-41", 2020-07-17T12:18:13.186 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.060681987Z", 2020-07-17T12:18:13.186 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.186 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.186 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.186 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.187 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.187 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_W8htjS8YlHsEASKVqW9GGew", 2020-07-17T12:18:13.187 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.187 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.188 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.188 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.188 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.188 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.299855_42.1", 2020-07-17T12:18:13.189 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.189 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-1:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.1", 2020-07-17T12:18:13.189 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.299855768Z", 2020-07-17T12:18:13.189 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.189 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.191 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fCYmrES933MNW-ZftW97tAH", 2020-07-17T12:18:13.191 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.191 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.192 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.192 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.318362_43.1", 2020-07-17T12:18:13.193 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.193 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-1:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.1", 2020-07-17T12:18:13.193 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.318362600Z", 2020-07-17T12:18:13.193 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.193 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.194 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.194 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.194 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.194 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_fCYmrES933MNW-ZftW97tAH", 2020-07-17T12:18:13.194 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.195 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.195 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.196 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.322473_44.1", 2020-07-17T12:18:13.196 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.196 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-1", 2020-07-17T12:18:13.197 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.322473615Z", 2020-07-17T12:18:13.197 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.197 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.197 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.197 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.198 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.198 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_N3sIyaABoPKhvVJDTubfANC", 2020-07-17T12:18:13.198 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.199 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.199 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.199 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.199 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.200 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.326365_45.1", 2020-07-17T12:18:13.200 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.200 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-1", 2020-07-17T12:18:13.200 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.326365120Z", 2020-07-17T12:18:13.200 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.201 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.201 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.201 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.201 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.201 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_N3sIyaABoPKhvVJDTubfANC", 2020-07-17T12:18:13.202 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.202 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.202 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.203 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.371051_46.1", 2020-07-17T12:18:13.203 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.204 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-2:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.2", 2020-07-17T12:18:13.204 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.371051231Z", 2020-07-17T12:18:13.204 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.204 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.204 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.205 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.205 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.205 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_bpM5KcuqzRHhKrpzuuAWOQK", 2020-07-17T12:18:13.205 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.206 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.206 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.207 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.376155_47.1", 2020-07-17T12:18:13.207 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.207 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-2:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.2", 2020-07-17T12:18:13.207 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.376155577Z", 2020-07-17T12:18:13.208 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.208 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.208 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.208 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.208 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.209 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_bpM5KcuqzRHhKrpzuuAWOQK", 2020-07-17T12:18:13.209 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.209 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.209 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.210 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.378460_48.1", 2020-07-17T12:18:13.211 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.211 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-2", 2020-07-17T12:18:13.211 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.378460095Z", 2020-07-17T12:18:13.212 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.212 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.212 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.213 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.213 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.214 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_o0AcCUXo7hxp6mur0vpSEA6", 2020-07-17T12:18:13.214 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.214 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.215 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.215 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.216 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987594.382370_49.1", 2020-07-17T12:18:13.216 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.216 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-2", 2020-07-17T12:18:13.216 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:34.382370235Z", 2020-07-17T12:18:13.217 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.217 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.217 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.217 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.218 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.218 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_o0AcCUXo7hxp6mur0vpSEA6", 2020-07-17T12:18:13.218 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.219 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.219 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.219 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.219 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.220 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987598.171972_50.1", 2020-07-17T12:18:13.220 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.220 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-3:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.3", 2020-07-17T12:18:13.220 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:38.171972137Z", 2020-07-17T12:18:13.221 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.221 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.221 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.221 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.222 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.222 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jXpmpU1bMKc6cP4eCcdC0E5", 2020-07-17T12:18:13.222 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.222 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.222 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.223 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987598.175455_51.1", 2020-07-17T12:18:13.224 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.224 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-3:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.3", 2020-07-17T12:18:13.224 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:38.175455638Z", 2020-07-17T12:18:13.224 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.224 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.225 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.226 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.226 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jXpmpU1bMKc6cP4eCcdC0E5", 2020-07-17T12:18:13.226 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.227 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.227 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.227 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.236 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987598.177003_52.1", 2020-07-17T12:18:13.236 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.236 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-3", 2020-07-17T12:18:13.237 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:38.177003803Z", 2020-07-17T12:18:13.237 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.237 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.237 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.238 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.238 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.238 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_-b65PCt1-zecno94kbqdZQV", 2020-07-17T12:18:13.238 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.239 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.239 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.239 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.239 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.240 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987598.180439_53.1", 2020-07-17T12:18:13.240 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.240 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-3", 2020-07-17T12:18:13.241 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:38.180439200Z", 2020-07-17T12:18:13.241 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.241 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.242 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.242 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.243 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.243 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_-b65PCt1-zecno94kbqdZQV", 2020-07-17T12:18:13.243 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.244 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.244 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.246 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.433011_54.1", 2020-07-17T12:18:13.246 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.246 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-4:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.4", 2020-07-17T12:18:13.246 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.433011959Z", 2020-07-17T12:18:13.247 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.247 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.247 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.248 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.248 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.249 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__ea-Segy13dxgNiJPXZ7Zd9", 2020-07-17T12:18:13.249 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.250 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.250 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.251 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.438333_55.1", 2020-07-17T12:18:13.252 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.252 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-4:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.4", 2020-07-17T12:18:13.253 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.438333452Z", 2020-07-17T12:18:13.253 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.253 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.254 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.254 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.255 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.255 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "__ea-Segy13dxgNiJPXZ7Zd9", 2020-07-17T12:18:13.255 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.256 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.256 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.257 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.258 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.440677_56.1", 2020-07-17T12:18:13.258 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.258 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-4", 2020-07-17T12:18:13.259 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.440677898Z", 2020-07-17T12:18:13.259 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.259 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.260 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.260 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.261 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.261 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_WRyLUNBmhHEnabp7o-DhlbH", 2020-07-17T12:18:13.261 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.262 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.262 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.264 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.445531_57.1", 2020-07-17T12:18:13.264 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.265 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-4", 2020-07-17T12:18:13.265 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.445531158Z", 2020-07-17T12:18:13.265 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.266 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.266 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.266 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.267 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.267 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_WRyLUNBmhHEnabp7o-DhlbH", 2020-07-17T12:18:13.267 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.268 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.268 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.270 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.468350_58.1", 2020-07-17T12:18:13.270 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.271 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-5:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.5", 2020-07-17T12:18:13.271 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.468350431Z", 2020-07-17T12:18:13.271 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.272 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.272 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.272 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.273 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.273 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_FMcLIdx8RNwH2qOVrLRBKLI", 2020-07-17T12:18:13.274 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.274 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.275 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.276 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.472556_59.1", 2020-07-17T12:18:13.277 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.277 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-5:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.5", 2020-07-17T12:18:13.277 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.472556928Z", 2020-07-17T12:18:13.278 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.278 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.279 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.279 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.280 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_FMcLIdx8RNwH2qOVrLRBKLI", 2020-07-17T12:18:13.280 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.281 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.281 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.282 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.474186_60.1", 2020-07-17T12:18:13.282 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.282 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-5", 2020-07-17T12:18:13.282 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.474186224Z", 2020-07-17T12:18:13.283 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.283 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.283 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.283 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.284 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.284 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_7oOlSD0kLzDgbQh79FmiwVN", 2020-07-17T12:18:13.284 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.284 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.284 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.285 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.477662_61.1", 2020-07-17T12:18:13.286 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.286 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-5", 2020-07-17T12:18:13.286 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.477662122Z", 2020-07-17T12:18:13.286 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.286 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.287 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.287 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.287 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.287 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_7oOlSD0kLzDgbQh79FmiwVN", 2020-07-17T12:18:13.287 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.288 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.288 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.288 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.288 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.289 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.518094_62.1", 2020-07-17T12:18:13.289 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.289 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-6:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.6", 2020-07-17T12:18:13.290 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.518094230Z", 2020-07-17T12:18:13.290 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.290 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.290 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.290 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.291 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.291 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_98i-mz1lveZeu2xWlqfjP8v", 2020-07-17T12:18:13.291 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.291 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.292 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.293 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.525483_63.1", 2020-07-17T12:18:13.293 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.293 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-6:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.6", 2020-07-17T12:18:13.293 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.525483008Z", 2020-07-17T12:18:13.293 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.293 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.294 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.294 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.294 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.294 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_98i-mz1lveZeu2xWlqfjP8v", 2020-07-17T12:18:13.295 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.295 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.295 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.296 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.527486_64.1", 2020-07-17T12:18:13.297 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.297 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-6", 2020-07-17T12:18:13.297 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.527486008Z", 2020-07-17T12:18:13.297 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.297 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.298 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.298 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.298 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.298 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6KmjjH6ijuVTEeEq7jAXi7H", 2020-07-17T12:18:13.298 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.299 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.299 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.300 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987599.533990_65.1", 2020-07-17T12:18:13.300 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.301 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-6", 2020-07-17T12:18:13.301 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:39.533990618Z", 2020-07-17T12:18:13.301 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.301 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.301 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.302 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.302 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.302 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6KmjjH6ijuVTEeEq7jAXi7H", 2020-07-17T12:18:13.302 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.303 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.303 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.304 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.314142_66.1", 2020-07-17T12:18:13.304 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.304 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.7", 2020-07-17T12:18:13.304 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.314142380Z", 2020-07-17T12:18:13.304 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.305 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.305 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.305 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.305 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.306 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_3p7wlqKX2jIqNSa48PpGSPB", 2020-07-17T12:18:13.306 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.306 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.306 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.307 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.321643_67.1", 2020-07-17T12:18:13.307 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.308 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.7", 2020-07-17T12:18:13.308 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.321643612Z", 2020-07-17T12:18:13.308 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.308 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.308 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.309 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.309 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.309 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_3p7wlqKX2jIqNSa48PpGSPB", 2020-07-17T12:18:13.309 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.310 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.310 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.311 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.324252_68.1", 2020-07-17T12:18:13.311 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.311 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7", 2020-07-17T12:18:13.311 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.324252955Z", 2020-07-17T12:18:13.312 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.312 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.312 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.312 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.312 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.313 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:13.313 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.313 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.313 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.314 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.329888_69.1", 2020-07-17T12:18:13.314 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.315 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7", 2020-07-17T12:18:13.315 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.329888896Z", 2020-07-17T12:18:13.315 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.315 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.315 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.316 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.316 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.316 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:13.316 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.317 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.317 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.318 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.360390_70.1", 2020-07-17T12:18:13.318 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.318 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.8", 2020-07-17T12:18:13.318 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.360390239Z", 2020-07-17T12:18:13.318 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.319 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.319 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.319 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.319 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.320 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r09s2aK_hqiAA214GJtpxF1", 2020-07-17T12:18:13.320 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.320 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.320 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.321 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.321 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.364581_71.1", 2020-07-17T12:18:13.321 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.322 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.8", 2020-07-17T12:18:13.322 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.364581063Z", 2020-07-17T12:18:13.322 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.322 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.322 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.322 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.323 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.323 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_r09s2aK_hqiAA214GJtpxF1", 2020-07-17T12:18:13.323 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.324 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.324 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.325 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.369486_72.1", 2020-07-17T12:18:13.325 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.325 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8", 2020-07-17T12:18:13.325 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.369486518Z", 2020-07-17T12:18:13.325 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.326 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.326 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.326 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.326 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.326 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:13.327 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.327 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.327 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.328 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.373949_73.1", 2020-07-17T12:18:13.328 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.329 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8", 2020-07-17T12:18:13.329 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.373949060Z", 2020-07-17T12:18:13.329 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.329 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.329 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.329 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.330 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.330 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:13.330 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.331 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.331 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.332 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.424920_75.1", 2020-07-17T12:18:13.332 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.332 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.9", 2020-07-17T12:18:13.332 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.424920583Z", 2020-07-17T12:18:13.332 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.333 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.333 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.333 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_znrNp-QgDthcyR2ClROORQe", 2020-07-17T12:18:13.334 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.334 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.334 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.335 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.431645_76.1", 2020-07-17T12:18:13.335 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.335 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.9", 2020-07-17T12:18:13.336 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.431645809Z", 2020-07-17T12:18:13.336 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.336 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.336 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.336 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.337 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.337 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_znrNp-QgDthcyR2ClROORQe", 2020-07-17T12:18:13.337 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.337 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.338 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.339 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.434319_77.1", 2020-07-17T12:18:13.339 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.339 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9", 2020-07-17T12:18:13.339 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.434319003Z", 2020-07-17T12:18:13.339 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.340 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.340 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.340 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.340 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.341 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:13.341 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.342 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.342 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:13.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.343 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987602.439076_78.1", 2020-07-17T12:18:13.343 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.343 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9", 2020-07-17T12:18:13.344 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:42.439076836Z", 2020-07-17T12:18:13.344 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.344 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.344 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:13.344 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:13.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.345 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.345 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:13.345 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.345 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.346 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.346 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.701989_80.1", 2020-07-17T12:18:13.347 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.347 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7", 2020-07-17T12:18:13.347 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.701989372Z", 2020-07-17T12:18:13.347 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.347 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.348 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:13.348 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.348 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.348 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:13.348 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.349 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.349 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "remove" 2020-07-17T12:18:13.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.350 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.708007_81.1", 2020-07-17T12:18:13.350 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.350 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-7", 2020-07-17T12:18:13.350 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.708007772Z", 2020-07-17T12:18:13.351 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.351 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.351 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:13.351 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.352 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.352 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:13.352 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.352 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.353 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.354 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.781097_82.1", 2020-07-17T12:18:13.354 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.354 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8", 2020-07-17T12:18:13.354 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.781097190Z", 2020-07-17T12:18:13.354 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.354 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.355 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:13.355 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.355 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.355 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:13.356 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.356 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.356 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "remove" 2020-07-17T12:18:13.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.357 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.786755_83.1", 2020-07-17T12:18:13.357 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.357 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-8", 2020-07-17T12:18:13.358 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.786755007Z", 2020-07-17T12:18:13.358 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.358 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.358 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:13.358 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.359 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.360 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:13.360 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.360 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.361 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.362 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.889801_85.1", 2020-07-17T12:18:13.362 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.362 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9", 2020-07-17T12:18:13.362 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.889801769Z", 2020-07-17T12:18:13.362 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.363 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.363 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:13.363 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.363 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.364 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:13.364 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.364 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.364 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "remove" 2020-07-17T12:18:13.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:13.365 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594987603.895710_86.1", 2020-07-17T12:18:13.365 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:13.366 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-9", 2020-07-17T12:18:13.366 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710719Z", 2020-07-17T12:18:13.366 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:13.366 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:13.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:13.366 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:13.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.367 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:13.367 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:13.367 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 2 2020-07-17T12:18:13.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:13.368 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:13.368 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:13.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:13.368 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:13.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:13.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:13.371 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 34954458-202e-48c3-864b-c00b15f41dd5 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:13.419 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:13.419 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:13.621 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:18:13.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.621 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987471.882144_44.1", 2020-07-17T12:18:13.621 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "user", 2020-07-17T12:18:13.622 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw-multisite-test-user", 2020-07-17T12:18:13.622 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.882144008Z", 2020-07-17T12:18:13.622 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.622 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.622 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.623 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.623 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.623 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_uJ2kpFkaIMR3kSeSIGfEB3T", 2020-07-17T12:18:13.623 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.624 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.624 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.624 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.624 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.625 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987471.890977_45.1", 2020-07-17T12:18:13.625 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "user", 2020-07-17T12:18:13.625 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw-multisite-test-user", 2020-07-17T12:18:13.625 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:04:31.890977275Z", 2020-07-17T12:18:13.626 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.626 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.626 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.626 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.627 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.627 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_uJ2kpFkaIMR3kSeSIGfEB3T", 2020-07-17T12:18:13.627 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.627 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.172466_48.1", 2020-07-17T12:18:13.629 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.629 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-10:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.10", 2020-07-17T12:18:13.629 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.172466793Z", 2020-07-17T12:18:13.629 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.629 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.630 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.630 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.630 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.630 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_mG7u6kOIVSSgWYLweK5uzvb", 2020-07-17T12:18:13.630 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.631 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.631 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.631 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.631 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.632 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.632 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.177556_49.1", 2020-07-17T12:18:13.632 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.632 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-10:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.10", 2020-07-17T12:18:13.633 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.177556387Z", 2020-07-17T12:18:13.633 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.633 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.633 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.633 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.634 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.634 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_mG7u6kOIVSSgWYLweK5uzvb", 2020-07-17T12:18:13.634 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.634 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.643 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.643 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.643 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.644 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.644 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.182179_50.1", 2020-07-17T12:18:13.644 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.644 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-10", 2020-07-17T12:18:13.644 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.182179376Z", 2020-07-17T12:18:13.645 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.645 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.645 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.645 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.646 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.646 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_yuMwADwMvlFMi5Y6H00iObr", 2020-07-17T12:18:13.646 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.646 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.647 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.647 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.647 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.647 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.647 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.186476_51.1", 2020-07-17T12:18:13.648 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.648 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-10", 2020-07-17T12:18:13.648 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.186476843Z", 2020-07-17T12:18:13.648 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.648 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.649 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.649 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.649 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.649 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_yuMwADwMvlFMi5Y6H00iObr", 2020-07-17T12:18:13.649 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.650 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.650 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.650 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.650 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.651 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.216713_52.1", 2020-07-17T12:18:13.651 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.651 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-11:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.11", 2020-07-17T12:18:13.652 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.216713690Z", 2020-07-17T12:18:13.652 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.652 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.652 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.652 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.653 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.653 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gq5MHLpwjP0npYemQW4217g", 2020-07-17T12:18:13.653 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.654 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.654 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.654 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.654 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.222092_53.1", 2020-07-17T12:18:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-11:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.11", 2020-07-17T12:18:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.222092496Z", 2020-07-17T12:18:13.655 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.656 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gq5MHLpwjP0npYemQW4217g", 2020-07-17T12:18:13.657 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.657 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.660 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.660 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.660 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.661 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.223732_54.1", 2020-07-17T12:18:13.661 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.661 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-11", 2020-07-17T12:18:13.661 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.223732073Z", 2020-07-17T12:18:13.662 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.662 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.662 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.662 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.662 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.663 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_hmDQZxZXEQLZ3GX5-Ia1DXm", 2020-07-17T12:18:13.663 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.663 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.663 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.664 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.664 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.664 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.664 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.227773_55.1", 2020-07-17T12:18:13.664 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.665 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-11", 2020-07-17T12:18:13.665 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.227773848Z", 2020-07-17T12:18:13.665 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.665 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.665 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.666 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.666 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.666 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_hmDQZxZXEQLZ3GX5-Ia1DXm", 2020-07-17T12:18:13.667 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.667 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.667 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.667 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.668 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.668 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.668 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.263759_56.1", 2020-07-17T12:18:13.668 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.669 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-12:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.12", 2020-07-17T12:18:13.669 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.263759779Z", 2020-07-17T12:18:13.669 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.669 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.669 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.670 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.670 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.670 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_kJPn3awfkoAAv2YoILjX1YX", 2020-07-17T12:18:13.670 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.671 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.671 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.671 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.671 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.671 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.672 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.272286_57.1", 2020-07-17T12:18:13.672 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.672 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-12:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.12", 2020-07-17T12:18:13.672 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.272286467Z", 2020-07-17T12:18:13.672 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.673 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.673 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.673 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.673 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.673 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_kJPn3awfkoAAv2YoILjX1YX", 2020-07-17T12:18:13.674 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.674 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.674 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.674 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.675 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.675 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.274467_58.1", 2020-07-17T12:18:13.675 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.675 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-12", 2020-07-17T12:18:13.676 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.274467332Z", 2020-07-17T12:18:13.676 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.676 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.676 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.676 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.677 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.677 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_WFmqiEkT2ufoxFeioom356B", 2020-07-17T12:18:13.677 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.677 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.678 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.678 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.678 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.679 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987605.278144_59.1", 2020-07-17T12:18:13.679 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.679 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-12", 2020-07-17T12:18:13.679 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:45.278144599Z", 2020-07-17T12:18:13.679 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.679 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.680 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.680 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.680 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.680 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_WFmqiEkT2ufoxFeioom356B", 2020-07-17T12:18:13.681 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.681 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.681 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.682 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.712640_60.1", 2020-07-17T12:18:13.682 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.682 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-13:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.13", 2020-07-17T12:18:13.683 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.712640382Z", 2020-07-17T12:18:13.683 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.683 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.683 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.683 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.684 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.684 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_coVsWkNqvKBUCWs8AacIoNl", 2020-07-17T12:18:13.684 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.684 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.685 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.685 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.717263_61.1", 2020-07-17T12:18:13.686 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.686 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-13:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.13", 2020-07-17T12:18:13.686 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.717263960Z", 2020-07-17T12:18:13.686 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.686 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.687 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.687 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.687 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.687 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_coVsWkNqvKBUCWs8AacIoNl", 2020-07-17T12:18:13.687 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.688 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.688 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.689 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.719823_62.1", 2020-07-17T12:18:13.689 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.689 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-13", 2020-07-17T12:18:13.689 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.719823082Z", 2020-07-17T12:18:13.690 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.690 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.690 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.690 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.691 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.691 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dWoVeSt6MXyutBS6iBOB_SN", 2020-07-17T12:18:13.691 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.691 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.691 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.692 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.692 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.725332_63.1", 2020-07-17T12:18:13.693 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.693 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-13", 2020-07-17T12:18:13.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.725332531Z", 2020-07-17T12:18:13.693 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.693 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.693 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.694 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.694 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.694 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dWoVeSt6MXyutBS6iBOB_SN", 2020-07-17T12:18:13.694 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.695 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.695 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.695 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.695 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.696 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.756675_64.1", 2020-07-17T12:18:13.696 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.696 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-14:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.14", 2020-07-17T12:18:13.696 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.756675726Z", 2020-07-17T12:18:13.697 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.697 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.697 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.697 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.697 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.698 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_BWPfn12CXl2TlU4AWhdAySv", 2020-07-17T12:18:13.698 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.698 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.698 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.699 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.761544_65.1", 2020-07-17T12:18:13.699 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.700 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-14:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.14", 2020-07-17T12:18:13.700 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.761544051Z", 2020-07-17T12:18:13.700 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.700 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.700 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.700 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.701 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.701 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_BWPfn12CXl2TlU4AWhdAySv", 2020-07-17T12:18:13.701 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.702 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.702 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.702 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.702 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.703 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.763526_66.1", 2020-07-17T12:18:13.703 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.703 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-14", 2020-07-17T12:18:13.703 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.763526945Z", 2020-07-17T12:18:13.703 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.704 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.704 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.704 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.704 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.704 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_irk8P5j0CkMCoEl6rP2eANQ", 2020-07-17T12:18:13.705 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.705 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.705 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.706 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.767394_67.1", 2020-07-17T12:18:13.706 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.707 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-14", 2020-07-17T12:18:13.707 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.767394556Z", 2020-07-17T12:18:13.707 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.707 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.707 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.707 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.708 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.708 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_irk8P5j0CkMCoEl6rP2eANQ", 2020-07-17T12:18:13.708 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.709 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.709 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.710 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.810791_68.1", 2020-07-17T12:18:13.710 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.710 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-15:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.15", 2020-07-17T12:18:13.710 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.810791998Z", 2020-07-17T12:18:13.710 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.711 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.711 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.711 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.711 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.711 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_aYP5BWVqE2ZQzg8KQUMoKIk", 2020-07-17T12:18:13.712 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.712 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.712 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.713 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.713 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.816286_69.1", 2020-07-17T12:18:13.713 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.713 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-15:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.15", 2020-07-17T12:18:13.714 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.816286566Z", 2020-07-17T12:18:13.714 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.714 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.714 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.714 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.715 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.715 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_aYP5BWVqE2ZQzg8KQUMoKIk", 2020-07-17T12:18:13.715 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.715 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.716 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.716 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.716 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.717 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.820905_70.1", 2020-07-17T12:18:13.717 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.717 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-15", 2020-07-17T12:18:13.717 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.820905407Z", 2020-07-17T12:18:13.717 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.717 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.718 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.718 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.718 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.718 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_rpYWMBKFNIdTSlnC72SKitx", 2020-07-17T12:18:13.719 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.719 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.719 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.719 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.719 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.720 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987618.830894_71.1", 2020-07-17T12:18:13.720 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.720 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-15", 2020-07-17T12:18:13.721 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:58.830894206Z", 2020-07-17T12:18:13.721 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.721 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.721 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.721 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.722 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.722 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_rpYWMBKFNIdTSlnC72SKitx", 2020-07-17T12:18:13.722 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.722 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.723 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.723 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.723 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.724 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.232376_72.1", 2020-07-17T12:18:13.724 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.724 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:13.724 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.232376434Z", 2020-07-17T12:18:13.724 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.724 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.725 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.725 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.725 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.725 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:13.726 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.726 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.726 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.726 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.726 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.727 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.236895_73.1", 2020-07-17T12:18:13.727 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.727 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:13.728 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.236895263Z", 2020-07-17T12:18:13.728 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.728 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.728 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.728 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.729 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.729 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:13.729 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.729 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.730 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.730 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.730 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.730 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.238681_74.1", 2020-07-17T12:18:13.731 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.731 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16", 2020-07-17T12:18:13.731 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.238681635Z", 2020-07-17T12:18:13.731 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.731 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.732 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.732 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.732 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.732 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_lEb4NYqynMKhYQrCh3ur9Pg", 2020-07-17T12:18:13.732 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.242366_75.1", 2020-07-17T12:18:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16", 2020-07-17T12:18:13.734 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.242366508Z", 2020-07-17T12:18:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_lEb4NYqynMKhYQrCh3ur9Pg", 2020-07-17T12:18:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.737 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.269805_76.1", 2020-07-17T12:18:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.269805039Z", 2020-07-17T12:18:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.738 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:13.739 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.741 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.273253_77.1", 2020-07-17T12:18:13.741 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.741 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:13.741 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.273253052Z", 2020-07-17T12:18:13.742 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.742 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.742 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.742 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.743 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.743 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:13.743 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.743 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.743 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.744 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.274824_78.1", 2020-07-17T12:18:13.745 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.745 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17", 2020-07-17T12:18:13.745 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.274824384Z", 2020-07-17T12:18:13.745 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.745 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.745 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.746 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.746 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.746 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_kC34LrBQXhenfmJdKuMdgnY", 2020-07-17T12:18:13.746 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.747 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.747 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.748 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.277982_79.1", 2020-07-17T12:18:13.748 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.748 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17", 2020-07-17T12:18:13.748 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.277982524Z", 2020-07-17T12:18:13.748 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.749 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.749 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.749 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.749 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.750 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_kC34LrBQXhenfmJdKuMdgnY", 2020-07-17T12:18:13.750 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.750 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.750 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.750 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.751 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.321344_80.1", 2020-07-17T12:18:13.751 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.752 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:13.752 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.321344326Z", 2020-07-17T12:18:13.752 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.752 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.752 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.752 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.753 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.753 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:13.753 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.754 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.754 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.754 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.754 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.755 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.325975_81.1", 2020-07-17T12:18:13.755 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.755 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:13.755 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.325975985Z", 2020-07-17T12:18:13.755 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.755 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.756 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.756 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.756 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.756 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:13.757 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.757 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.757 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.758 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.758 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.327813_82.1", 2020-07-17T12:18:13.758 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.758 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18", 2020-07-17T12:18:13.759 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.327813080Z", 2020-07-17T12:18:13.759 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.759 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.759 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.759 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.760 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.760 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_0c-EMQmUZNyncsOLFYYvBXZ", 2020-07-17T12:18:13.760 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.760 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.761 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.761 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.331302_83.1", 2020-07-17T12:18:13.762 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.762 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18", 2020-07-17T12:18:13.762 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.331302239Z", 2020-07-17T12:18:13.762 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.762 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.763 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.763 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.763 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.763 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_0c-EMQmUZNyncsOLFYYvBXZ", 2020-07-17T12:18:13.763 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.764 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.764 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.764 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.764 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.765 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.349661_84.1", 2020-07-17T12:18:13.765 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.765 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:13.765 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.349661555Z", 2020-07-17T12:18:13.766 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.766 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.766 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:13.766 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.766 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.767 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:13.767 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.767 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.767 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.354642_85.1", 2020-07-17T12:18:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.769 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-16:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.16", 2020-07-17T12:18:13.769 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.354642695Z", 2020-07-17T12:18:13.769 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.769 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.769 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oZ6M7p0VdIaoGAta7MmONm5", 2020-07-17T12:18:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.771 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.771 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.358809_86.1", 2020-07-17T12:18:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.358809552Z", 2020-07-17T12:18:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.775 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.364350_87.1", 2020-07-17T12:18:13.775 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-17:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.17", 2020-07-17T12:18:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.364350896Z", 2020-07-17T12:18:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.777 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.777 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_dylRgxyYfuQn_-97zX1VnMh", 2020-07-17T12:18:13.777 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.374842_88.1", 2020-07-17T12:18:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.374842199Z", 2020-07-17T12:18:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.780 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:13.780 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.780 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.780 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:13.781 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.781 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.781 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.782 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987653.381436_89.1", 2020-07-17T12:18:13.782 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.782 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-18:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.18", 2020-07-17T12:18:13.783 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:07:33.381436296Z", 2020-07-17T12:18:13.783 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.783 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.783 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:13.783 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.784 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.784 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__8Bb3bd7aprS_unn-1ZY4lt", 2020-07-17T12:18:13.784 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.784 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.785 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.785 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.785 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.785 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.785 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.580189_90.1", 2020-07-17T12:18:13.786 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.786 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:13.786 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.580189016Z", 2020-07-17T12:18:13.786 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.786 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.787 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.787 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.787 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.787 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:13.787 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.788 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.788 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.788 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.788 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.789 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.583928_91.1", 2020-07-17T12:18:13.789 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.789 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:13.789 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.583928438Z", 2020-07-17T12:18:13.790 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.790 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.790 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.790 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.791 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.791 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:13.791 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.791 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.791 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.792 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.792 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.792 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.585595_92.1", 2020-07-17T12:18:13.793 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.793 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19", 2020-07-17T12:18:13.793 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.585595449Z", 2020-07-17T12:18:13.793 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.793 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.793 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.794 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.794 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.794 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Y0anvog_R4e6DQW9utotNY_", 2020-07-17T12:18:13.794 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.795 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.795 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.795 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.795 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.796 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.589056_93.1", 2020-07-17T12:18:13.796 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.796 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19", 2020-07-17T12:18:13.796 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.589056210Z", 2020-07-17T12:18:13.797 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.797 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.797 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.797 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.797 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.798 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Y0anvog_R4e6DQW9utotNY_", 2020-07-17T12:18:13.798 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.798 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.799 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.799 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.799 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.799 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.800 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.596888_94.1", 2020-07-17T12:18:13.800 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.800 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:13.800 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.596888308Z", 2020-07-17T12:18:13.800 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.801 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.801 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:13.801 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.801 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.801 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:13.802 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.802 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.802 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.802 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987843.602473_95.1", 2020-07-17T12:18:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.803 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-19:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.19", 2020-07-17T12:18:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:43.602473621Z", 2020-07-17T12:18:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:13.804 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Vogm8H67NTTz89DdLyJwOvo", 2020-07-17T12:18:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.805 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987998.537142_96.1", 2020-07-17T12:18:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.537142107Z", 2020-07-17T12:18:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.807 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.808 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987998.541926_97.1", 2020-07-17T12:18:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.541926282Z", 2020-07-17T12:18:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.811 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.812 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987998.544300_98.1", 2020-07-17T12:18:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20", 2020-07-17T12:18:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.544300322Z", 2020-07-17T12:18:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.814 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.815 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_qoKh-ps-xZklh5sIuAPFkjZ", 2020-07-17T12:18:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987998.548993_99.1", 2020-07-17T12:18:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.817 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20", 2020-07-17T12:18:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:18.548993097Z", 2020-07-17T12:18:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_qoKh-ps-xZklh5sIuAPFkjZ", 2020-07-17T12:18:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.819 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.820 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988004.926158_100.1", 2020-07-17T12:18:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:24.926158716Z", 2020-07-17T12:18:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.821 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.822 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.823 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988004.936521_101.1", 2020-07-17T12:18:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:13.824 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:13:24.936521959Z", 2020-07-17T12:18:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.826 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.827 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988065.046888_102.1", 2020-07-17T12:18:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:25.046888654Z", 2020-07-17T12:18:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.828 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988065.056306_103.1", 2020-07-17T12:18:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-20:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.20", 2020-07-17T12:18:13.831 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:14:25.056306412Z", 2020-07-17T12:18:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.832 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zNiuvMasax7-QKsEr39vhRY", 2020-07-17T12:18:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.833 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.834 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.108561_114.1", 2020-07-17T12:18:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.108561553Z", 2020-07-17T12:18:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.835 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:13.836 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.837 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.846 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.112561_115.1", 2020-07-17T12:18:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.112561600Z", 2020-07-17T12:18:13.847 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.848 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.114341_116.1", 2020-07-17T12:18:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21", 2020-07-17T12:18:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.114341527Z", 2020-07-17T12:18:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.851 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_7IHshvdvzhTN_iaS63UlhhP", 2020-07-17T12:18:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.118095_117.1", 2020-07-17T12:18:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21", 2020-07-17T12:18:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.118095935Z", 2020-07-17T12:18:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_7IHshvdvzhTN_iaS63UlhhP", 2020-07-17T12:18:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.144536_119.1", 2020-07-17T12:18:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.144536481Z", 2020-07-17T12:18:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.148697_120.1", 2020-07-17T12:18:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.148697190Z", 2020-07-17T12:18:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.150760_121.1", 2020-07-17T12:18:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22", 2020-07-17T12:18:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.150760146Z", 2020-07-17T12:18:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gsPtnlqFRKAZ04Ktvzs95UN", 2020-07-17T12:18:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.866 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.155597_122.1", 2020-07-17T12:18:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22", 2020-07-17T12:18:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.155597648Z", 2020-07-17T12:18:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gsPtnlqFRKAZ04Ktvzs95UN", 2020-07-17T12:18:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.195741_123.1", 2020-07-17T12:18:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.195741697Z", 2020-07-17T12:18:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.873 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.200019_124.1", 2020-07-17T12:18:13.874 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.200019763Z", 2020-07-17T12:18:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.875 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.201967_125.1", 2020-07-17T12:18:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23", 2020-07-17T12:18:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.201967975Z", 2020-07-17T12:18:13.878 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.879 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_4EWRUD1pbHkY-FiDIwWfNaQ", 2020-07-17T12:18:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.880 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.205751_126.1", 2020-07-17T12:18:13.881 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23", 2020-07-17T12:18:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.205751555Z", 2020-07-17T12:18:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.882 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_4EWRUD1pbHkY-FiDIwWfNaQ", 2020-07-17T12:18:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.883 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.226616_127.1", 2020-07-17T12:18:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.226616279Z", 2020-07-17T12:18:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.885 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.886 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.231665_128.1", 2020-07-17T12:18:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.888 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-21:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.21", 2020-07-17T12:18:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.231665701Z", 2020-07-17T12:18:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_g-j-oKLl26liwpZNjT7CiWF", 2020-07-17T12:18:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.890 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.236032_130.1", 2020-07-17T12:18:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.236032642Z", 2020-07-17T12:18:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.892 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:13.893 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.894 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.240206_131.1", 2020-07-17T12:18:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-22:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.22", 2020-07-17T12:18:13.895 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.240206565Z", 2020-07-17T12:18:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1bNig1ucJW6S3Tr0xu0qRXq", 2020-07-17T12:18:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.897 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.898 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.251296_132.1", 2020-07-17T12:18:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.251296772Z", 2020-07-17T12:18:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.899 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:13.900 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988109.262832_133.1", 2020-07-17T12:18:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-23:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.23", 2020-07-17T12:18:13.902 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:09.262832933Z", 2020-07-17T12:18:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_pDCH6I7aiHBqZ4CMvQtgx-V", 2020-07-17T12:18:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.904 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.905 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.216266_162.1", 2020-07-17T12:18:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.216266874Z", 2020-07-17T12:18:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.906 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.907 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.220314_163.1", 2020-07-17T12:18:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:13.909 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.220314597Z", 2020-07-17T12:18:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.910 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.911 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.222492_164.1", 2020-07-17T12:18:13.912 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24", 2020-07-17T12:18:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.222492473Z", 2020-07-17T12:18:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.913 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1aTUtnO5v7_BzTHJ7dcWax9", 2020-07-17T12:18:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.226266_165.1", 2020-07-17T12:18:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24", 2020-07-17T12:18:13.916 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.226266565Z", 2020-07-17T12:18:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1aTUtnO5v7_BzTHJ7dcWax9", 2020-07-17T12:18:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.918 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.254586_166.1", 2020-07-17T12:18:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.254586031Z", 2020-07-17T12:18:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.920 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.921 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.922 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.922 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.922 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.923 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.923 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.259320_167.1", 2020-07-17T12:18:13.923 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.923 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:13.923 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.259320320Z", 2020-07-17T12:18:13.924 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.924 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.924 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.924 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.924 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.925 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.925 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.925 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.925 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.925 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.926 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.926 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.261985_168.1", 2020-07-17T12:18:13.926 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.927 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25", 2020-07-17T12:18:13.927 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.261985157Z", 2020-07-17T12:18:13.927 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.927 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.927 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.928 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.928 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.928 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_CJpPPaq5s3R_DLWK7_Dw6N9", 2020-07-17T12:18:13.928 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.929 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.929 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.929 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.929 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.930 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.268842_169.1", 2020-07-17T12:18:13.930 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.930 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25", 2020-07-17T12:18:13.930 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.268842944Z", 2020-07-17T12:18:13.930 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.931 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.931 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.931 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.931 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.932 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_CJpPPaq5s3R_DLWK7_Dw6N9", 2020-07-17T12:18:13.932 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.932 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.932 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.933 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.933 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.310784_170.1", 2020-07-17T12:18:13.933 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.934 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:13.934 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.310784639Z", 2020-07-17T12:18:13.934 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.934 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.934 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.934 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.935 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.935 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.935 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.936 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.936 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.936 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.936 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.937 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.319433_171.1", 2020-07-17T12:18:13.937 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.937 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:13.937 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.319433996Z", 2020-07-17T12:18:13.937 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.938 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.938 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.938 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.938 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.322090_172.1", 2020-07-17T12:18:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26", 2020-07-17T12:18:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.322090964Z", 2020-07-17T12:18:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_OI4YC_rV8-IvJU_ONTZMZ22", 2020-07-17T12:18:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.326100_173.1", 2020-07-17T12:18:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26", 2020-07-17T12:18:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.326100789Z", 2020-07-17T12:18:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_OI4YC_rV8-IvJU_ONTZMZ22", 2020-07-17T12:18:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.345795_174.1", 2020-07-17T12:18:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.345795397Z", 2020-07-17T12:18:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.950 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.950 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.950 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.950 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.351197_175.1", 2020-07-17T12:18:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.351197071Z", 2020-07-17T12:18:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.952 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.952 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.952 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.952 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.953 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.953 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.953 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.953 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.953 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.954 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.354716_176.1", 2020-07-17T12:18:13.954 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.954 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:13.955 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.354716106Z", 2020-07-17T12:18:13.955 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.955 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.955 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.955 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.956 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.956 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.956 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:13.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.956 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.957 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.957 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.957 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.957 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.360435_177.1", 2020-07-17T12:18:13.958 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.958 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-24:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.24", 2020-07-17T12:18:13.958 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.360435248Z", 2020-07-17T12:18:13.958 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.958 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.959 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.959 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.959 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.959 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_j-egXpayH84_HZFx2_jf3zx", 2020-07-17T12:18:13.960 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:13.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.960 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.960 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.960 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.960 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.961 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.961 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.365795_178.1", 2020-07-17T12:18:13.961 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.961 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:13.962 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.365795374Z", 2020-07-17T12:18:13.962 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.962 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.962 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.962 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.963 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.963 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.963 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.963 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.964 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.964 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.964 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.964 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.964 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.372555_179.1", 2020-07-17T12:18:13.965 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.965 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:13.965 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.372555105Z", 2020-07-17T12:18:13.965 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.965 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.966 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.966 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.966 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.966 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.966 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.967 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.967 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.967 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.967 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.968 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.385196_180.1", 2020-07-17T12:18:13.968 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.968 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:13.968 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.385196882Z", 2020-07-17T12:18:13.969 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.969 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.969 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.969 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.970 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.970 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.970 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:13.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.970 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.970 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.971 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.971 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.971 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.971 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.393417_181.1", 2020-07-17T12:18:13.971 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.972 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-25:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.25", 2020-07-17T12:18:13.972 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.393417880Z", 2020-07-17T12:18:13.972 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.972 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.972 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.973 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.973 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.973 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NXymIHLW6ESb2c6zUndfMZs", 2020-07-17T12:18:13.973 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:13.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.974 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.974 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.974 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.974 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.975 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.405697_182.1", 2020-07-17T12:18:13.975 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.975 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:13.975 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.405697153Z", 2020-07-17T12:18:13.976 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.976 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.976 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.976 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.976 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.977 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.977 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.977 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.977 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.978 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.978 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.978 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.415076_183.1", 2020-07-17T12:18:13.978 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.979 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:13.979 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.415076707Z", 2020-07-17T12:18:13.979 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.979 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.979 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.979 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.980 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.980 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.980 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.981 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.981 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.982 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.426554_184.1", 2020-07-17T12:18:13.982 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.982 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:13.982 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.426554025Z", 2020-07-17T12:18:13.982 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.983 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.983 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.983 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.983 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.983 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.984 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:13.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.984 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.984 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.984 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.985 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.985 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988126.433699_185.1", 2020-07-17T12:18:13.985 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.985 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-26:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.26", 2020-07-17T12:18:13.986 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:26.433699189Z", 2020-07-17T12:18:13.986 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.986 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.986 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.986 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:13.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.987 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.987 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_1KKz7XXD0_ti4gRfg24m4py", 2020-07-17T12:18:13.987 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 3 2020-07-17T12:18:13.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.987 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.988 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.988 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.988 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.989 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.192609_186.1", 2020-07-17T12:18:13.989 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.989 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:13.989 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.192609023Z", 2020-07-17T12:18:13.989 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.990 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.990 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.990 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.990 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.990 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:13.991 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.991 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.991 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.991 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.992 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.992 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.197302_187.1", 2020-07-17T12:18:13.992 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:13.992 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:13.993 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.197302293Z", 2020-07-17T12:18:13.993 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.993 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.993 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.993 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.994 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.994 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:13.994 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.994 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.995 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:13.995 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.995 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.995 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.995 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.200146_188.1", 2020-07-17T12:18:13.996 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.996 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27", 2020-07-17T12:18:13.996 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.200146111Z", 2020-07-17T12:18:13.996 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:13.996 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:13.997 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:13.997 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:13.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.997 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:13.997 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_EAM3oic4BNiOLUongn3X0x8", 2020-07-17T12:18:13.997 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:13.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.998 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:13.998 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:13.998 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.998 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:13.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:13.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:13.999 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.204265_189.1", 2020-07-17T12:18:13.999 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:13.999 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27", 2020-07-17T12:18:13.999 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.204265093Z", 2020-07-17T12:18:14.000 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.000 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.000 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.000 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.001 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.001 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_EAM3oic4BNiOLUongn3X0x8", 2020-07-17T12:18:14.001 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.001 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.001 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.002 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.002 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.002 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.002 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.232604_190.1", 2020-07-17T12:18:14.003 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.003 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:14.003 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.232604283Z", 2020-07-17T12:18:14.003 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.003 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.003 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.004 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.004 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.004 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:14.004 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.005 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.005 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.005 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.005 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.006 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.236653_191.1", 2020-07-17T12:18:14.006 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.006 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:14.006 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.236653061Z", 2020-07-17T12:18:14.007 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.007 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.007 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.007 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.007 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.008 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:14.008 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.008 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.008 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.009 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.009 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.009 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.238308_192.1", 2020-07-17T12:18:14.009 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.010 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28", 2020-07-17T12:18:14.010 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.238308219Z", 2020-07-17T12:18:14.010 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.010 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.010 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.011 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.011 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.011 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_8tICQSSPhT4sUiRts2CGtXF", 2020-07-17T12:18:14.012 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.012 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.012 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.012 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.012 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.013 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.242121_193.1", 2020-07-17T12:18:14.013 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.013 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28", 2020-07-17T12:18:14.014 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.242121826Z", 2020-07-17T12:18:14.014 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.014 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.014 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.014 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.015 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.015 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_8tICQSSPhT4sUiRts2CGtXF", 2020-07-17T12:18:14.015 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.015 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.015 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.016 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.016 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.016 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.291427_194.1", 2020-07-17T12:18:14.017 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.017 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:14.017 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.291427038Z", 2020-07-17T12:18:14.017 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.017 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.017 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.018 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.018 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.018 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:14.018 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.019 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.019 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.020 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.296400_195.1", 2020-07-17T12:18:14.020 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.020 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:14.020 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.296400519Z", 2020-07-17T12:18:14.021 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.021 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.021 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.021 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.021 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.022 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:14.022 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.022 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.022 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.022 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.023 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.023 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.299100_196.1", 2020-07-17T12:18:14.023 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.024 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29", 2020-07-17T12:18:14.024 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.299100799Z", 2020-07-17T12:18:14.024 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.024 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.024 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.024 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.025 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.025 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oUHOV9TdQ-W-rj70_Cou9X5", 2020-07-17T12:18:14.025 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.026 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.026 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.027 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.304751_197.1", 2020-07-17T12:18:14.027 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.027 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29", 2020-07-17T12:18:14.027 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.304751236Z", 2020-07-17T12:18:14.027 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oUHOV9TdQ-W-rj70_Cou9X5", 2020-07-17T12:18:14.029 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.029 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.029 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.029 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.329636_198.1", 2020-07-17T12:18:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.329636096Z", 2020-07-17T12:18:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.334331_199.1", 2020-07-17T12:18:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-27:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.27", 2020-07-17T12:18:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.334331520Z", 2020-07-17T12:18:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_MO80p5R0wLomdi2T7R_Dyr0", 2020-07-17T12:18:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.388479_200.1", 2020-07-17T12:18:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.388479756Z", 2020-07-17T12:18:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.398272_201.1", 2020-07-17T12:18:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-28:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.28", 2020-07-17T12:18:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.398272953Z", 2020-07-17T12:18:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zM83_T8JxqmBLmgVAyTywYC", 2020-07-17T12:18:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.043 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.043 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.051 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.051 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.052 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.463878_202.1", 2020-07-17T12:18:14.052 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.052 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:14.052 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.463878801Z", 2020-07-17T12:18:14.052 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.053 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.053 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:14.053 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.053 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.054 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:14.054 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.054 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.054 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.474476_203.1", 2020-07-17T12:18:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-29:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.29", 2020-07-17T12:18:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.474476699Z", 2020-07-17T12:18:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_JL75tqb8u2-CnPWfwESHy8X", 2020-07-17T12:18:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.543823_204.1", 2020-07-17T12:18:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.543823468Z", 2020-07-17T12:18:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.062 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.062 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.548616_205.1", 2020-07-17T12:18:14.062 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.063 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:14.063 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.548616662Z", 2020-07-17T12:18:14.063 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.063 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.063 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.065 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.065 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.550563_206.1", 2020-07-17T12:18:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30", 2020-07-17T12:18:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.550563904Z", 2020-07-17T12:18:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.067 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.067 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.067 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.067 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_lJ2S3jRGkcD2jZEcb0bU-NY", 2020-07-17T12:18:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.069 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.555228_207.1", 2020-07-17T12:18:14.069 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.070 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30", 2020-07-17T12:18:14.070 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.555228975Z", 2020-07-17T12:18:14.070 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.070 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.070 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.070 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.071 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.071 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_lJ2S3jRGkcD2jZEcb0bU-NY", 2020-07-17T12:18:14.071 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.072 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.072 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.073 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.579434_208.1", 2020-07-17T12:18:14.073 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.073 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:14.073 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.579434456Z", 2020-07-17T12:18:14.073 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.074 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.074 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.074 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.074 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.075 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:14.075 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.075 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.075 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.076 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.076 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.583820_209.1", 2020-07-17T12:18:14.076 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.077 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:14.077 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.583820075Z", 2020-07-17T12:18:14.077 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.077 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.077 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.077 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.078 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.078 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:14.078 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.079 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.079 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.080 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.585899_210.1", 2020-07-17T12:18:14.080 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.080 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31", 2020-07-17T12:18:14.080 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.585899826Z", 2020-07-17T12:18:14.080 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.081 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.081 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.081 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.081 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.081 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_ln1eOUq8lDkVI4IpQvkNAi4", 2020-07-17T12:18:14.082 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.082 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.082 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.082 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.083 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.589417_211.1", 2020-07-17T12:18:14.083 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.083 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31", 2020-07-17T12:18:14.084 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.589417398Z", 2020-07-17T12:18:14.084 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.084 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.084 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.084 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.085 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.085 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_ln1eOUq8lDkVI4IpQvkNAi4", 2020-07-17T12:18:14.085 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.085 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.086 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.086 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.638248_212.1", 2020-07-17T12:18:14.087 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.087 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:14.087 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.638248979Z", 2020-07-17T12:18:14.087 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.087 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.088 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.088 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.088 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.088 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:14.088 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.089 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.089 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.090 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.642956_213.1", 2020-07-17T12:18:14.090 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.090 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:14.090 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.642956157Z", 2020-07-17T12:18:14.091 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.091 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.091 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.091 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.092 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.092 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:14.092 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.092 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.092 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.093 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.644834_214.1", 2020-07-17T12:18:14.093 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.094 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32", 2020-07-17T12:18:14.094 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.644834814Z", 2020-07-17T12:18:14.094 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.094 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.094 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.095 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.095 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.095 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_CTGkGtU8NTFdGu11wP4HBlE", 2020-07-17T12:18:14.095 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.096 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.096 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.097 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.649548_215.1", 2020-07-17T12:18:14.097 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.097 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32", 2020-07-17T12:18:14.097 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.649548275Z", 2020-07-17T12:18:14.097 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.098 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.098 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.098 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.098 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.099 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_CTGkGtU8NTFdGu11wP4HBlE", 2020-07-17T12:18:14.099 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.099 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.099 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.100 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.100 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.100 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.691484_216.1", 2020-07-17T12:18:14.100 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.100 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:14.101 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.691484060Z", 2020-07-17T12:18:14.101 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.101 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.101 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:14.101 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.102 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.102 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:14.102 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.102 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.103 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.103 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.104 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.695812_217.1", 2020-07-17T12:18:14.104 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.104 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-30:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.30", 2020-07-17T12:18:14.104 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.695812573Z", 2020-07-17T12:18:14.104 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.105 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.105 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:14.105 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.105 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.105 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fTtwgRv27stH7XIkGDVSNtz", 2020-07-17T12:18:14.106 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.106 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.106 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.107 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.107 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.752305_218.1", 2020-07-17T12:18:14.107 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.107 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:14.108 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.752305251Z", 2020-07-17T12:18:14.108 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.108 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.108 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:14.108 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.109 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.109 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:14.109 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.109 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.110 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.110 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.759930_219.1", 2020-07-17T12:18:14.111 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.111 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-31:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.31", 2020-07-17T12:18:14.111 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.759930328Z", 2020-07-17T12:18:14.111 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.111 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.112 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:14.112 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.112 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.112 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r06_H0O8q-7fTjwP1s-FIiA", 2020-07-17T12:18:14.113 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.113 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.113 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.113 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.113 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.114 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.819822_220.1", 2020-07-17T12:18:14.114 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.114 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:14.114 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.819822109Z", 2020-07-17T12:18:14.115 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.115 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.115 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:14.115 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.116 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.116 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:14.116 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.116 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.116 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.117 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.117 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.117 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.829887_221.1", 2020-07-17T12:18:14.118 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.118 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-32:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.32", 2020-07-17T12:18:14.118 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.829887197Z", 2020-07-17T12:18:14.118 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.118 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.118 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:14.119 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.119 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.119 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_XTVxPYOo49-Ol7fYlPk5NPN", 2020-07-17T12:18:14.119 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.120 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.120 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.121 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.121 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.897711_222.1", 2020-07-17T12:18:14.121 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.121 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:14.121 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.897711412Z", 2020-07-17T12:18:14.121 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.122 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.122 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.122 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.122 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.123 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:14.123 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.123 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.123 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.124 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.124 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.901139_223.1", 2020-07-17T12:18:14.124 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.125 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:14.125 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.901139991Z", 2020-07-17T12:18:14.125 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.125 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.125 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.125 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.126 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.126 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:14.126 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.126 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.127 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.127 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.127 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.128 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.902679_224.1", 2020-07-17T12:18:14.128 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.128 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33", 2020-07-17T12:18:14.128 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.902679722Z", 2020-07-17T12:18:14.128 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.128 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.129 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.129 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.129 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.129 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_UxWO-19hsf0zfhgE2d7C0Bs", 2020-07-17T12:18:14.130 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.130 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.130 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.131 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.906270_225.1", 2020-07-17T12:18:14.131 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.131 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33", 2020-07-17T12:18:14.132 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.906270781Z", 2020-07-17T12:18:14.132 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.132 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.132 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.132 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.133 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.133 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_UxWO-19hsf0zfhgE2d7C0Bs", 2020-07-17T12:18:14.133 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.133 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.134 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.134 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.932251_226.1", 2020-07-17T12:18:14.135 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.135 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:14.135 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.932251093Z", 2020-07-17T12:18:14.135 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.135 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.136 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.136 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.136 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.136 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:14.136 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.137 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.137 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.137 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.137 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.138 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.138 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.937747_227.1", 2020-07-17T12:18:14.138 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.138 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:14.138 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.937747244Z", 2020-07-17T12:18:14.139 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.139 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.139 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.139 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.139 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.140 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:14.140 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.140 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.140 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.141 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.939489_228.1", 2020-07-17T12:18:14.141 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.142 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34", 2020-07-17T12:18:14.142 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.939489887Z", 2020-07-17T12:18:14.142 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.142 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.142 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.143 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.143 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.143 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oKhT8xShmiPT7_3DwLLaiZf", 2020-07-17T12:18:14.143 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.144 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.144 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.145 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.942587_229.1", 2020-07-17T12:18:14.145 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.145 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34", 2020-07-17T12:18:14.145 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.942587543Z", 2020-07-17T12:18:14.145 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.146 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.146 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.146 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.146 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.146 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_oKhT8xShmiPT7_3DwLLaiZf", 2020-07-17T12:18:14.147 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.147 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.147 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.148 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.977552_230.1", 2020-07-17T12:18:14.148 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.148 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:14.149 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.977552760Z", 2020-07-17T12:18:14.149 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.149 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.149 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.149 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.150 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.150 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:14.150 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.150 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.151 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.151 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.152 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.982605_231.1", 2020-07-17T12:18:14.152 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.152 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:14.152 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.982605704Z", 2020-07-17T12:18:14.152 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.152 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.153 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.153 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.153 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.153 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:14.153 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.154 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.154 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.154 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.154 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.155 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.985248_232.1", 2020-07-17T12:18:14.155 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.155 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35", 2020-07-17T12:18:14.155 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.985248699Z", 2020-07-17T12:18:14.156 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.156 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.156 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.156 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.157 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.157 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_vxH0cpq2NUCS2-F5AwiLlGy", 2020-07-17T12:18:14.157 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.157 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.157 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.158 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.158 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.158 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.158 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988148.989554_233.1", 2020-07-17T12:18:14.159 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.159 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35", 2020-07-17T12:18:14.159 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:48.989554870Z", 2020-07-17T12:18:14.159 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.159 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.159 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.160 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.160 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.160 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_vxH0cpq2NUCS2-F5AwiLlGy", 2020-07-17T12:18:14.160 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.161 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.161 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.161 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.162 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.162 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988149.009800_234.1", 2020-07-17T12:18:14.162 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.162 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:14.162 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:49.009800265Z", 2020-07-17T12:18:14.163 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.163 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.163 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:14.163 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.163 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.164 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:14.164 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.164 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.164 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.164 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.165 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.165 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.165 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988149.014525_235.1", 2020-07-17T12:18:14.165 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.166 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-33:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.33", 2020-07-17T12:18:14.166 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:49.014525305Z", 2020-07-17T12:18:14.166 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.166 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.166 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:14.166 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.167 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.167 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_05qTDIiirzi2zJomy7PdraM", 2020-07-17T12:18:14.167 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.168 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.168 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.169 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988150.064957_236.1", 2020-07-17T12:18:14.169 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.169 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:14.169 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:50.064957041Z", 2020-07-17T12:18:14.169 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.169 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.170 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:14.170 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.170 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.170 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:14.171 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.171 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.171 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.172 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.172 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988150.071666_237.1", 2020-07-17T12:18:14.172 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.172 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-34:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.34", 2020-07-17T12:18:14.173 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:50.071666653Z", 2020-07-17T12:18:14.173 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.173 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.173 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:14.173 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.174 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.174 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__gWAQUubZwFXTadcxKhkoJc", 2020-07-17T12:18:14.174 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.174 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.181 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.182 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.128899_238.1", 2020-07-17T12:18:14.182 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.182 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:14.183 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.128899145Z", 2020-07-17T12:18:14.183 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.183 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.183 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:14.183 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.184 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.184 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:14.184 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.184 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.185 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.185 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.185 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.186 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.136044_239.1", 2020-07-17T12:18:14.186 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.186 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-35:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.35", 2020-07-17T12:18:14.186 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.136044802Z", 2020-07-17T12:18:14.186 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.186 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.187 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:14.187 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.187 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.187 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_se0mL3Ay31SluzmyiWbJ8rp", 2020-07-17T12:18:14.188 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.188 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.188 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.189 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.223773_240.1", 2020-07-17T12:18:14.189 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.189 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-36:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.36", 2020-07-17T12:18:14.189 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.223773094Z", 2020-07-17T12:18:14.190 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.190 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.190 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.190 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.191 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.191 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jdX6lkwwXVktMAEuRsLdL9f", 2020-07-17T12:18:14.191 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.191 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.191 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.192 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.227459_241.1", 2020-07-17T12:18:14.193 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.193 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-36:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.36", 2020-07-17T12:18:14.193 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.227459296Z", 2020-07-17T12:18:14.193 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.193 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.193 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.194 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.194 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.194 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jdX6lkwwXVktMAEuRsLdL9f", 2020-07-17T12:18:14.194 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.195 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.195 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.195 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.195 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.196 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.229174_242.1", 2020-07-17T12:18:14.196 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.196 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-36", 2020-07-17T12:18:14.196 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.229174371Z", 2020-07-17T12:18:14.197 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.197 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.197 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.197 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.197 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_qgrqKYBWaNJyRRn3Vjx0wVN", 2020-07-17T12:18:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.198 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.199 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.199 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.232729_243.1", 2020-07-17T12:18:14.199 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.200 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-36", 2020-07-17T12:18:14.200 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.232729435Z", 2020-07-17T12:18:14.200 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.200 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.200 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.200 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.201 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.201 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_qgrqKYBWaNJyRRn3Vjx0wVN", 2020-07-17T12:18:14.201 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.202 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.202 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.202 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.202 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.202 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.203 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.269955_244.1", 2020-07-17T12:18:14.203 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.203 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-37:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.37", 2020-07-17T12:18:14.203 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.269955795Z", 2020-07-17T12:18:14.203 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.203 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.204 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.204 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.204 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.204 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_c-tGp8YfI6B1yBpFUhIj44i", 2020-07-17T12:18:14.205 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.205 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.205 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.206 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.206 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.275803_245.1", 2020-07-17T12:18:14.206 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.207 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-37:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.37", 2020-07-17T12:18:14.207 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.275803241Z", 2020-07-17T12:18:14.207 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.207 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.207 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.208 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.208 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.208 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_c-tGp8YfI6B1yBpFUhIj44i", 2020-07-17T12:18:14.208 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.209 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.209 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.209 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.210 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.278648_246.1", 2020-07-17T12:18:14.210 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.210 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-37", 2020-07-17T12:18:14.210 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.278648356Z", 2020-07-17T12:18:14.210 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.211 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.211 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.211 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.211 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.212 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_9F_FGhwZ3EnXzZglCBoTwNA", 2020-07-17T12:18:14.212 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.212 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.212 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.213 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.285189_247.1", 2020-07-17T12:18:14.213 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.213 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-37", 2020-07-17T12:18:14.214 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.285189703Z", 2020-07-17T12:18:14.214 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.214 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.214 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.214 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.215 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.215 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_9F_FGhwZ3EnXzZglCBoTwNA", 2020-07-17T12:18:14.215 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.215 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.216 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.216 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.216 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.217 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.334180_248.1", 2020-07-17T12:18:14.217 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.217 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-38:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.38", 2020-07-17T12:18:14.217 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.334180801Z", 2020-07-17T12:18:14.217 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.217 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.218 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.218 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_hVnb1ngvk17nkYrOuLaCqwM", 2020-07-17T12:18:14.219 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.219 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.219 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.220 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.337583_249.1", 2020-07-17T12:18:14.220 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.220 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-38:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.38", 2020-07-17T12:18:14.221 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.337583158Z", 2020-07-17T12:18:14.221 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.221 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.221 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.221 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.222 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.222 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_hVnb1ngvk17nkYrOuLaCqwM", 2020-07-17T12:18:14.222 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.222 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.223 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.223 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.223 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.223 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.339086_250.1", 2020-07-17T12:18:14.224 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.224 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-38", 2020-07-17T12:18:14.224 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.339086250Z", 2020-07-17T12:18:14.224 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.224 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.225 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.225 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.225 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.225 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_3EGy4bfasMLEwsmYrKxh2v5", 2020-07-17T12:18:14.225 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.226 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.226 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.227 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988151.342294_251.1", 2020-07-17T12:18:14.227 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.227 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-38", 2020-07-17T12:18:14.227 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:51.342294371Z", 2020-07-17T12:18:14.228 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.228 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.228 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.228 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.228 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.229 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_3EGy4bfasMLEwsmYrKxh2v5", 2020-07-17T12:18:14.229 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.230 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.230 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.231 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.949522_252.1", 2020-07-17T12:18:14.231 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.231 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-39:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.39", 2020-07-17T12:18:14.231 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.949522585Z", 2020-07-17T12:18:14.232 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.232 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.232 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.232 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.233 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.233 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gsVvIhPoNrpxq4YxIZghI20", 2020-07-17T12:18:14.233 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.233 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.234 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.235 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.956039_253.1", 2020-07-17T12:18:14.235 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.235 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-39:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.39", 2020-07-17T12:18:14.235 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.956039623Z", 2020-07-17T12:18:14.235 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.236 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.236 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.236 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.236 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.237 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_gsVvIhPoNrpxq4YxIZghI20", 2020-07-17T12:18:14.237 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.237 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.237 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.238 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.238 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.961169_254.1", 2020-07-17T12:18:14.239 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.239 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-39", 2020-07-17T12:18:14.239 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.961169072Z", 2020-07-17T12:18:14.239 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.239 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.240 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.240 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.240 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.240 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jHFRYvjwFaeKft8jqkFw9f6", 2020-07-17T12:18:14.240 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.241 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.241 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.242 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.965532_255.1", 2020-07-17T12:18:14.242 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.242 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-39", 2020-07-17T12:18:14.243 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.965532718Z", 2020-07-17T12:18:14.243 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.243 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.243 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.243 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.244 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.244 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jHFRYvjwFaeKft8jqkFw9f6", 2020-07-17T12:18:14.244 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.244 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.245 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.245 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.245 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.246 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988152.996911_256.1", 2020-07-17T12:18:14.246 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.246 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-40:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.40", 2020-07-17T12:18:14.246 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.996911090Z", 2020-07-17T12:18:14.246 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.246 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.247 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.247 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.247 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.247 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zd9daTaLaaQPrwIpjVSY9v4", 2020-07-17T12:18:14.248 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.248 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.248 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.248 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.249 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.001607_257.1", 2020-07-17T12:18:14.249 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.250 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-40:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.40", 2020-07-17T12:18:14.250 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.001607160Z", 2020-07-17T12:18:14.250 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.250 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.250 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.251 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.251 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.251 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zd9daTaLaaQPrwIpjVSY9v4", 2020-07-17T12:18:14.251 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.252 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.252 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.252 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.252 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.253 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.004358_258.1", 2020-07-17T12:18:14.253 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.253 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-40", 2020-07-17T12:18:14.253 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.004358398Z", 2020-07-17T12:18:14.254 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.254 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.254 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.254 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.255 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.255 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Apkycxzmtorr0IIi3gzU-Qm", 2020-07-17T12:18:14.255 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.255 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.256 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.256 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.256 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.256 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.009492_259.1", 2020-07-17T12:18:14.257 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.257 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-40", 2020-07-17T12:18:14.257 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.009492644Z", 2020-07-17T12:18:14.257 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.257 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.258 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.258 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.258 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.258 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Apkycxzmtorr0IIi3gzU-Qm", 2020-07-17T12:18:14.259 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.259 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.259 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.259 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.260 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.048556_260.1", 2020-07-17T12:18:14.260 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.261 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-41:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.41", 2020-07-17T12:18:14.261 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.048556572Z", 2020-07-17T12:18:14.261 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.261 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.261 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.261 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.262 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.262 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NntBaMu1tW2iuZqebtOm9OP", 2020-07-17T12:18:14.262 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.263 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.263 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.264 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.053032_261.1", 2020-07-17T12:18:14.264 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.264 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-41:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.41", 2020-07-17T12:18:14.264 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.053032571Z", 2020-07-17T12:18:14.265 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.265 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.265 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.265 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.265 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.266 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_NntBaMu1tW2iuZqebtOm9OP", 2020-07-17T12:18:14.266 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.266 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.266 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.267 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.267 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.267 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.267 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.055326_262.1", 2020-07-17T12:18:14.268 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.268 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-41", 2020-07-17T12:18:14.268 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.055326711Z", 2020-07-17T12:18:14.268 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.268 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_W8htjS8YlHsEASKVqW9GGew", 2020-07-17T12:18:14.270 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.270 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.270 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.270 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.271 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.271 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988153.060681_263.1", 2020-07-17T12:18:14.271 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.271 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-41", 2020-07-17T12:18:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.060681987Z", 2020-07-17T12:18:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.273 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.273 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_W8htjS8YlHsEASKVqW9GGew", 2020-07-17T12:18:14.273 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.275 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.299855_42.1", 2020-07-17T12:18:14.275 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.275 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-1:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.1", 2020-07-17T12:18:14.275 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.299855768Z", 2020-07-17T12:18:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fCYmrES933MNW-ZftW97tAH", 2020-07-17T12:18:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.279 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.318362_43.1", 2020-07-17T12:18:14.279 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.279 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-1:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.1", 2020-07-17T12:18:14.279 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.318362600Z", 2020-07-17T12:18:14.279 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.281 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_fCYmrES933MNW-ZftW97tAH", 2020-07-17T12:18:14.281 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.281 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.281 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.282 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.282 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.282 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.322473_44.1", 2020-07-17T12:18:14.283 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.283 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-1", 2020-07-17T12:18:14.283 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.322473615Z", 2020-07-17T12:18:14.283 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.283 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.284 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.284 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.284 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.284 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_N3sIyaABoPKhvVJDTubfANC", 2020-07-17T12:18:14.285 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.285 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.285 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.286 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.286 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.326365_45.1", 2020-07-17T12:18:14.286 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.286 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-1", 2020-07-17T12:18:14.287 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.326365120Z", 2020-07-17T12:18:14.287 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.287 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.287 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.287 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.288 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.288 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_N3sIyaABoPKhvVJDTubfANC", 2020-07-17T12:18:14.288 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.289 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.289 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.289 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.289 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.290 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.290 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.371051_46.1", 2020-07-17T12:18:14.290 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.290 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-2:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.2", 2020-07-17T12:18:14.290 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.371051231Z", 2020-07-17T12:18:14.290 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.291 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.291 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.291 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.291 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.292 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_bpM5KcuqzRHhKrpzuuAWOQK", 2020-07-17T12:18:14.292 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.292 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.292 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.292 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.293 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.293 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.376155_47.1", 2020-07-17T12:18:14.293 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.294 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-2:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.2", 2020-07-17T12:18:14.294 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.376155577Z", 2020-07-17T12:18:14.294 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.294 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.294 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.295 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.295 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.295 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_bpM5KcuqzRHhKrpzuuAWOQK", 2020-07-17T12:18:14.295 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.296 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.296 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.296 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.296 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.297 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.378460_48.1", 2020-07-17T12:18:14.297 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.297 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-2", 2020-07-17T12:18:14.297 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.378460095Z", 2020-07-17T12:18:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_o0AcCUXo7hxp6mur0vpSEA6", 2020-07-17T12:18:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.301 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987594.382370_49.1", 2020-07-17T12:18:14.301 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.301 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-2", 2020-07-17T12:18:14.301 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:34.382370235Z", 2020-07-17T12:18:14.301 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.302 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.302 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.302 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.302 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.302 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_o0AcCUXo7hxp6mur0vpSEA6", 2020-07-17T12:18:14.303 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.303 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.303 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.303 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.304 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.304 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.304 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987598.171972_50.1", 2020-07-17T12:18:14.304 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.305 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-3:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.3", 2020-07-17T12:18:14.305 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.171972137Z", 2020-07-17T12:18:14.305 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.305 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.305 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.306 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.306 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.306 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jXpmpU1bMKc6cP4eCcdC0E5", 2020-07-17T12:18:14.306 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.307 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.307 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.308 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987598.175455_51.1", 2020-07-17T12:18:14.308 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.308 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-3:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.3", 2020-07-17T12:18:14.309 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.175455638Z", 2020-07-17T12:18:14.309 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.309 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.309 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.309 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.310 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.310 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jXpmpU1bMKc6cP4eCcdC0E5", 2020-07-17T12:18:14.310 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.310 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.311 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.312 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987598.177003_52.1", 2020-07-17T12:18:14.312 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.312 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-3", 2020-07-17T12:18:14.312 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.177003803Z", 2020-07-17T12:18:14.312 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.313 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.313 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.313 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.313 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.314 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_-b65PCt1-zecno94kbqdZQV", 2020-07-17T12:18:14.314 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.314 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.314 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.315 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.315 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.315 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.315 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987598.180439_53.1", 2020-07-17T12:18:14.315 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.316 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-3", 2020-07-17T12:18:14.316 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:38.180439200Z", 2020-07-17T12:18:14.316 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.316 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.316 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.317 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.317 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.317 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_-b65PCt1-zecno94kbqdZQV", 2020-07-17T12:18:14.317 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.318 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.318 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.319 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.433011_54.1", 2020-07-17T12:18:14.319 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.319 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-4:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.4", 2020-07-17T12:18:14.319 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.433011959Z", 2020-07-17T12:18:14.320 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.320 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.320 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.320 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.321 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.321 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__ea-Segy13dxgNiJPXZ7Zd9", 2020-07-17T12:18:14.321 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.322 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.322 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.323 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.438333_55.1", 2020-07-17T12:18:14.323 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.323 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-4:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.4", 2020-07-17T12:18:14.323 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.438333452Z", 2020-07-17T12:18:14.323 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.324 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.324 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.325 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.325 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.325 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "__ea-Segy13dxgNiJPXZ7Zd9", 2020-07-17T12:18:14.325 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.326 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.326 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.327 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.440677_56.1", 2020-07-17T12:18:14.327 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.327 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-4", 2020-07-17T12:18:14.327 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.440677898Z", 2020-07-17T12:18:14.328 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.328 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.328 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.328 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.329 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.329 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_WRyLUNBmhHEnabp7o-DhlbH", 2020-07-17T12:18:14.329 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.329 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.330 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.330 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.445531_57.1", 2020-07-17T12:18:14.331 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.331 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-4", 2020-07-17T12:18:14.331 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.445531158Z", 2020-07-17T12:18:14.331 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.331 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.332 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.332 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.332 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.332 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_WRyLUNBmhHEnabp7o-DhlbH", 2020-07-17T12:18:14.332 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.333 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.333 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.333 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.333 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.334 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.334 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.468350_58.1", 2020-07-17T12:18:14.334 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.334 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-5:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.5", 2020-07-17T12:18:14.335 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.468350431Z", 2020-07-17T12:18:14.335 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.335 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.335 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.335 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.336 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.336 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_FMcLIdx8RNwH2qOVrLRBKLI", 2020-07-17T12:18:14.336 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.336 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.337 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.337 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.337 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.338 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.338 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.472556_59.1", 2020-07-17T12:18:14.339 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.339 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-5:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.5", 2020-07-17T12:18:14.340 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.472556928Z", 2020-07-17T12:18:14.340 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.340 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.340 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.341 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.341 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.341 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_FMcLIdx8RNwH2qOVrLRBKLI", 2020-07-17T12:18:14.341 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.342 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.342 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.343 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.343 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.474186_60.1", 2020-07-17T12:18:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.345 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-5", 2020-07-17T12:18:14.345 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.474186224Z", 2020-07-17T12:18:14.346 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.346 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.346 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.347 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.347 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.348 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_7oOlSD0kLzDgbQh79FmiwVN", 2020-07-17T12:18:14.348 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.349 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.349 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.349 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.350 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.351 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.477662_61.1", 2020-07-17T12:18:14.351 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.352 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-5", 2020-07-17T12:18:14.352 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.477662122Z", 2020-07-17T12:18:14.353 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.353 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.353 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.354 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.354 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.355 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_7oOlSD0kLzDgbQh79FmiwVN", 2020-07-17T12:18:14.355 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.356 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.356 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.356 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.358 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.518094_62.1", 2020-07-17T12:18:14.358 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.359 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-6:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.6", 2020-07-17T12:18:14.359 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.518094230Z", 2020-07-17T12:18:14.360 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.360 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.360 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.361 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.361 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.362 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_98i-mz1lveZeu2xWlqfjP8v", 2020-07-17T12:18:14.362 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.363 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.363 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.365 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.525483_63.1", 2020-07-17T12:18:14.365 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.366 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-6:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.6", 2020-07-17T12:18:14.366 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.525483008Z", 2020-07-17T12:18:14.367 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.367 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.367 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.368 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.368 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.369 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_98i-mz1lveZeu2xWlqfjP8v", 2020-07-17T12:18:14.369 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.370 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.370 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.371 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.372 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.527486_64.1", 2020-07-17T12:18:14.372 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.372 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-6", 2020-07-17T12:18:14.373 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.527486008Z", 2020-07-17T12:18:14.373 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.374 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.374 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.375 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.375 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.376 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6KmjjH6ijuVTEeEq7jAXi7H", 2020-07-17T12:18:14.376 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.377 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.377 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.377 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.378 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.378 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.379 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987599.533990_65.1", 2020-07-17T12:18:14.379 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.379 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-6", 2020-07-17T12:18:14.380 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:39.533990618Z", 2020-07-17T12:18:14.380 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.380 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.381 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.381 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.381 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.383 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.384 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6KmjjH6ijuVTEeEq7jAXi7H", 2020-07-17T12:18:14.384 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.385 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.385 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.386 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.386 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.387 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.314142_66.1", 2020-07-17T12:18:14.388 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.389 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.7", 2020-07-17T12:18:14.389 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.314142380Z", 2020-07-17T12:18:14.389 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.390 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.390 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.390 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.391 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.391 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_3p7wlqKX2jIqNSa48PpGSPB", 2020-07-17T12:18:14.392 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.392 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.393 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.393 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.394 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.394 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.395 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.321643_67.1", 2020-07-17T12:18:14.395 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.396 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.7", 2020-07-17T12:18:14.396 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.321643612Z", 2020-07-17T12:18:14.397 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.397 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.397 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.398 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.398 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.399 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.399 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_3p7wlqKX2jIqNSa48PpGSPB", 2020-07-17T12:18:14.399 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.400 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.401 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.401 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.401 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.402 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.324252_68.1", 2020-07-17T12:18:14.403 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.403 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7", 2020-07-17T12:18:14.403 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.324252955Z", 2020-07-17T12:18:14.404 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.404 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.404 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.405 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.405 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.406 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:14.406 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.406 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.407 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.407 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.407 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.409 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.329888_69.1", 2020-07-17T12:18:14.409 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.409 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7", 2020-07-17T12:18:14.409 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.329888896Z", 2020-07-17T12:18:14.410 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.410 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.410 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.411 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.411 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.411 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:14.411 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.412 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.412 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.412 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.412 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.413 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.360390_70.1", 2020-07-17T12:18:14.413 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.413 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.8", 2020-07-17T12:18:14.414 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.360390239Z", 2020-07-17T12:18:14.414 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.414 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.414 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.414 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.415 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.415 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r09s2aK_hqiAA214GJtpxF1", 2020-07-17T12:18:14.415 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.415 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.416 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.416 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.416 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.416 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.364581_71.1", 2020-07-17T12:18:14.417 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.417 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.8", 2020-07-17T12:18:14.417 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.364581063Z", 2020-07-17T12:18:14.417 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.417 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.418 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.418 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.418 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.418 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_r09s2aK_hqiAA214GJtpxF1", 2020-07-17T12:18:14.418 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.419 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.419 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.419 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.419 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.420 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.420 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.369486_72.1", 2020-07-17T12:18:14.420 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.420 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8", 2020-07-17T12:18:14.420 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.369486518Z", 2020-07-17T12:18:14.421 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.421 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.421 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.421 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.422 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.422 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:14.422 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.423 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.423 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.423 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.423 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.423 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.424 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.373949_73.1", 2020-07-17T12:18:14.424 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.424 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8", 2020-07-17T12:18:14.424 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.373949060Z", 2020-07-17T12:18:14.424 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.425 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.425 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.425 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.425 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.425 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:14.426 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.426 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.426 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.426 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.427 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.427 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.424920_75.1", 2020-07-17T12:18:14.427 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.427 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.9", 2020-07-17T12:18:14.428 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.424920583Z", 2020-07-17T12:18:14.428 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.428 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.428 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.428 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.429 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.429 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_znrNp-QgDthcyR2ClROORQe", 2020-07-17T12:18:14.429 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.430 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.430 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.431 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.431645_76.1", 2020-07-17T12:18:14.431 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.431 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9:bc359d1d-af48-401e-ac43-335f84c3f1d9.4466.9", 2020-07-17T12:18:14.431 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.431645809Z", 2020-07-17T12:18:14.431 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.432 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.432 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.432 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.433 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.433 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_znrNp-QgDthcyR2ClROORQe", 2020-07-17T12:18:14.434 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.434 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.435 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.436 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.434319_77.1", 2020-07-17T12:18:14.437 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.437 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9", 2020-07-17T12:18:14.437 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.434319003Z", 2020-07-17T12:18:14.437 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.438 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.438 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.438 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.438 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.439 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:14.439 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.439 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.439 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.440 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.440 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987602.439076_78.1", 2020-07-17T12:18:14.441 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.441 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9", 2020-07-17T12:18:14.441 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:42.439076836Z", 2020-07-17T12:18:14.441 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.442 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.442 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.442 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.442 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.443 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:14.443 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.443 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.443 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.444 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.444 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.444 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.701989_80.1", 2020-07-17T12:18:14.445 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.445 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7", 2020-07-17T12:18:14.445 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.701989372Z", 2020-07-17T12:18:14.445 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.445 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.446 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:14.446 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.446 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.446 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:14.447 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.447 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.447 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "remove" 2020-07-17T12:18:14.447 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.448 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.448 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.448 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.708007_81.1", 2020-07-17T12:18:14.448 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.448 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-7", 2020-07-17T12:18:14.449 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.708007772Z", 2020-07-17T12:18:14.449 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.449 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.449 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:14.450 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.451 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.451 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_IWxto9d3eJ8z88JHGLzHXX4", 2020-07-17T12:18:14.451 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.451 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.452 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.453 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.781097_82.1", 2020-07-17T12:18:14.453 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.453 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8", 2020-07-17T12:18:14.454 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.781097190Z", 2020-07-17T12:18:14.454 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.454 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.454 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:14.454 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.455 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.455 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:14.455 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.455 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.456 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "remove" 2020-07-17T12:18:14.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.457 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.786755_83.1", 2020-07-17T12:18:14.457 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.457 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-8", 2020-07-17T12:18:14.457 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.786755007Z", 2020-07-17T12:18:14.457 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.458 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.458 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:14.458 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.458 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.460 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_6bEwJX7H55B6iYx7Tt6KaB3", 2020-07-17T12:18:14.460 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.460 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.460 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.461 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.889801_85.1", 2020-07-17T12:18:14.462 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.462 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9", 2020-07-17T12:18:14.462 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.889801769Z", 2020-07-17T12:18:14.462 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.462 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.462 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:14.463 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.463 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.463 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:14.463 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.464 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.464 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "remove" 2020-07-17T12:18:14.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.465 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.465 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594987603.895710_86.1", 2020-07-17T12:18:14.465 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:14.465 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-9", 2020-07-17T12:18:14.465 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710719Z", 2020-07-17T12:18:14.466 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.466 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.466 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:14.466 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.467 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.467 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KhzN-TCPoe_PxPntkLePPnZ", 2020-07-17T12:18:14.467 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 2 2020-07-17T12:18:14.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.468 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.468 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.468 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.468 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.469 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.469 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:18:14.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:14.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:14.472 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 159da1b7-e55f-4be6-b11f-00c3d4a2d701 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:14.511 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:14.512 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:14.706 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988220.956172_305.1", 2020-07-17T12:18:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-42:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.1", 2020-07-17T12:18:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:00.956172510Z", 2020-07-17T12:18:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:14.707 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:14.708 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_89GWemIuo59MkTHrsZ96fDB", 2020-07-17T12:18:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988220.960732_311.1", 2020-07-17T12:18:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.710 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-42:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.1", 2020-07-17T12:18:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:00.960732737Z", 2020-07-17T12:18:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:14.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_89GWemIuo59MkTHrsZ96fDB", 2020-07-17T12:18:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.712 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988220.962740_312.1", 2020-07-17T12:18:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:14.714 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-42", 2020-07-17T12:18:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:00.962740344Z", 2020-07-17T12:18:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:14.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jd0QN7dKqOEvwhgkjExxHZJ", 2020-07-17T12:18:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.716 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:14.717 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988220.966697_313.1", 2020-07-17T12:18:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-42", 2020-07-17T12:18:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:00.966697652Z", 2020-07-17T12:18:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:14.718 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jd0QN7dKqOEvwhgkjExxHZJ", 2020-07-17T12:18:14.719 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988221.014026_314.1", 2020-07-17T12:18:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.721 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-43:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.2", 2020-07-17T12:18:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:01.014026413Z", 2020-07-17T12:18:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:14.722 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_ml0QmT8abPJvszGTM7ZTmMU", 2020-07-17T12:18:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:14.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988221.018339_315.1", 2020-07-17T12:18:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-43:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.2", 2020-07-17T12:18:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:01.018339644Z", 2020-07-17T12:18:14.725 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.726 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_ml0QmT8abPJvszGTM7ZTmMU", 2020-07-17T12:18:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:14.727 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:14.728 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988221.020094_316.1", 2020-07-17T12:18:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-43", 2020-07-17T12:18:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:01.020094867Z", 2020-07-17T12:18:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:14.729 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_cjc4jkibkd0CrkLsm8EzQTF", 2020-07-17T12:18:14.730 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988221.023402_317.1", 2020-07-17T12:18:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-43", 2020-07-17T12:18:14.732 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:01.023402058Z", 2020-07-17T12:18:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:14.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_cjc4jkibkd0CrkLsm8EzQTF", 2020-07-17T12:18:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:14.734 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:14.738 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:14.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:14.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:14.741 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 159da1b7-e55f-4be6-b11f-00c3d4a2d701 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:14.780 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:14.781 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:14.989 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:18:14.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.989 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988220.956172_305.1", 2020-07-17T12:18:14.990 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.990 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-42:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.1", 2020-07-17T12:18:14.990 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:00.956172510Z", 2020-07-17T12:18:14.990 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.991 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.992 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_89GWemIuo59MkTHrsZ96fDB", 2020-07-17T12:18:14.992 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.992 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988220.960732_311.1", 2020-07-17T12:18:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-42:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.1", 2020-07-17T12:18:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:00.960732737Z", 2020-07-17T12:18:14.994 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:14.995 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:14.995 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:14.998 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:14.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.998 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:14.998 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_89GWemIuo59MkTHrsZ96fDB", 2020-07-17T12:18:14.999 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:14.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:14.999 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:14.999 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:14.999 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.000 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988220.962740_312.1", 2020-07-17T12:18:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-42", 2020-07-17T12:18:15.001 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:00.962740344Z", 2020-07-17T12:18:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jd0QN7dKqOEvwhgkjExxHZJ", 2020-07-17T12:18:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988220.966697_313.1", 2020-07-17T12:18:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-42", 2020-07-17T12:18:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:00.966697652Z", 2020-07-17T12:18:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jd0QN7dKqOEvwhgkjExxHZJ", 2020-07-17T12:18:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.008 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:15.008 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.008 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.009 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988221.014026_314.1", 2020-07-17T12:18:15.009 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.009 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-43:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.2", 2020-07-17T12:18:15.009 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:01.014026413Z", 2020-07-17T12:18:15.009 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.010 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.010 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.010 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.010 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.011 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_ml0QmT8abPJvszGTM7ZTmMU", 2020-07-17T12:18:15.011 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.011 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.012 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:15.012 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.012 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.014 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988221.018339_315.1", 2020-07-17T12:18:15.014 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.014 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-43:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.2", 2020-07-17T12:18:15.015 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:01.018339644Z", 2020-07-17T12:18:15.015 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.016 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.016 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.016 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.017 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.017 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_ml0QmT8abPJvszGTM7ZTmMU", 2020-07-17T12:18:15.017 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.018 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.018 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:15.018 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.018 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.019 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988221.020094_316.1", 2020-07-17T12:18:15.020 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:15.020 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-43", 2020-07-17T12:18:15.020 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:01.020094867Z", 2020-07-17T12:18:15.021 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.021 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.021 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.022 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.022 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.023 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_cjc4jkibkd0CrkLsm8EzQTF", 2020-07-17T12:18:15.023 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.024 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.024 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:15.024 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.025 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988221.023402_317.1", 2020-07-17T12:18:15.026 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:15.026 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-43", 2020-07-17T12:18:15.027 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:01.023402058Z", 2020-07-17T12:18:15.027 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.027 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.028 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.028 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.029 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.029 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_cjc4jkibkd0CrkLsm8EzQTF", 2020-07-17T12:18:15.029 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.030 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.032 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:15.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.033 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:18:15.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:15.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:15.035 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 159da1b7-e55f-4be6-b11f-00c3d4a2d701 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:15.075 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:15.075 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:15.276 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:15.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.276 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988220.956172_305.1", 2020-07-17T12:18:15.277 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.277 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-42:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.1", 2020-07-17T12:18:15.277 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:00.956172510Z", 2020-07-17T12:18:15.277 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.277 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.278 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.278 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_89GWemIuo59MkTHrsZ96fDB", 2020-07-17T12:18:15.280 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.280 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.281 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:15.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.282 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988220.960732_311.1", 2020-07-17T12:18:15.282 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.282 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-42:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.1", 2020-07-17T12:18:15.282 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:00.960732737Z", 2020-07-17T12:18:15.282 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.283 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.283 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.283 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.283 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.283 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_89GWemIuo59MkTHrsZ96fDB", 2020-07-17T12:18:15.284 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.284 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.284 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:15.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.285 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988220.962740_312.1", 2020-07-17T12:18:15.285 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:15.288 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-42", 2020-07-17T12:18:15.289 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:00.962740344Z", 2020-07-17T12:18:15.289 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.289 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.289 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.290 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.290 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.290 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jd0QN7dKqOEvwhgkjExxHZJ", 2020-07-17T12:18:15.290 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.291 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.291 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:15.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.292 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988220.966697_313.1", 2020-07-17T12:18:15.292 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:15.293 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-42", 2020-07-17T12:18:15.293 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:00.966697652Z", 2020-07-17T12:18:15.293 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.293 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.293 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.294 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.294 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.294 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_jd0QN7dKqOEvwhgkjExxHZJ", 2020-07-17T12:18:15.294 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.295 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.296 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:15.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.297 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988221.014026_314.1", 2020-07-17T12:18:15.297 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.298 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-43:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.2", 2020-07-17T12:18:15.298 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:01.014026413Z", 2020-07-17T12:18:15.298 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.298 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.298 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.299 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.299 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.299 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_ml0QmT8abPJvszGTM7ZTmMU", 2020-07-17T12:18:15.299 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.300 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.300 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:15.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.301 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988221.018339_315.1", 2020-07-17T12:18:15.301 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.302 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-43:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.2", 2020-07-17T12:18:15.302 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:01.018339644Z", 2020-07-17T12:18:15.302 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.303 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.303 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.303 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.304 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.304 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_ml0QmT8abPJvszGTM7ZTmMU", 2020-07-17T12:18:15.304 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.304 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.305 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:15.305 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.305 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.306 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988221.020094_316.1", 2020-07-17T12:18:15.306 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:15.306 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-43", 2020-07-17T12:18:15.306 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:01.020094867Z", 2020-07-17T12:18:15.307 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.307 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.307 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.307 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.308 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.308 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_cjc4jkibkd0CrkLsm8EzQTF", 2020-07-17T12:18:15.308 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.308 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.309 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:15.309 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.309 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.309 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988221.023402_317.1", 2020-07-17T12:18:15.310 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:15.310 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-43", 2020-07-17T12:18:15.310 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:17:01.023402058Z", 2020-07-17T12:18:15.310 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.310 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.311 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.311 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_cjc4jkibkd0CrkLsm8EzQTF", 2020-07-17T12:18:15.311 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.312 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.313 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:15.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.314 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:15.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:15.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:15.315 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 159da1b7-e55f-4be6-b11f-00c3d4a2d701 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:15.353 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:15.354 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:15.558 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:18:15.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.559 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988220.956172_305.1", 2020-07-17T12:18:15.559 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.559 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-42:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.1", 2020-07-17T12:18:15.560 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:00.956172510Z", 2020-07-17T12:18:15.560 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.560 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.560 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.560 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.561 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.561 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_89GWemIuo59MkTHrsZ96fDB", 2020-07-17T12:18:15.561 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.562 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.562 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:15.562 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.562 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.563 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988220.960732_311.1", 2020-07-17T12:18:15.563 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.563 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-42:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.1", 2020-07-17T12:18:15.563 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:00.960732737Z", 2020-07-17T12:18:15.563 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.564 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.564 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.564 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.564 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.565 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_89GWemIuo59MkTHrsZ96fDB", 2020-07-17T12:18:15.565 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.565 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.565 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:15.565 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.566 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988220.962740_312.1", 2020-07-17T12:18:15.566 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:15.567 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-42", 2020-07-17T12:18:15.567 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:00.962740344Z", 2020-07-17T12:18:15.567 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.567 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.567 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.567 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.568 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.568 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jd0QN7dKqOEvwhgkjExxHZJ", 2020-07-17T12:18:15.568 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.569 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.569 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:15.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.570 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988220.966697_313.1", 2020-07-17T12:18:15.570 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:15.570 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-42", 2020-07-17T12:18:15.571 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:00.966697652Z", 2020-07-17T12:18:15.571 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.571 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.571 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.571 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.572 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.572 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_jd0QN7dKqOEvwhgkjExxHZJ", 2020-07-17T12:18:15.572 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.572 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.573 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:15.573 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.573 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.573 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.574 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988221.014026_314.1", 2020-07-17T12:18:15.574 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.574 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-43:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.2", 2020-07-17T12:18:15.574 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:01.014026413Z", 2020-07-17T12:18:15.574 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.574 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.575 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.575 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.575 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.575 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_ml0QmT8abPJvszGTM7ZTmMU", 2020-07-17T12:18:15.576 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.576 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.576 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:15.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.577 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988221.018339_315.1", 2020-07-17T12:18:15.577 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.577 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-43:f2444b35-3e93-4fbd-99ef-52c98595b68f.6508.2", 2020-07-17T12:18:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:01.018339644Z", 2020-07-17T12:18:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.579 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.579 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_ml0QmT8abPJvszGTM7ZTmMU", 2020-07-17T12:18:15.579 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.579 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988221.020094_316.1", 2020-07-17T12:18:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-43", 2020-07-17T12:18:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:01.020094867Z", 2020-07-17T12:18:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_cjc4jkibkd0CrkLsm8EzQTF", 2020-07-17T12:18:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.583 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.583 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:15.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988221.023402_317.1", 2020-07-17T12:18:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-43", 2020-07-17T12:18:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:17:01.023402058Z", 2020-07-17T12:18:15.585 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:15.585 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:15.585 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:15.585 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:15.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.586 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:15.586 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_cjc4jkibkd0CrkLsm8EzQTF", 2020-07-17T12:18:15.586 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:15.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:15.586 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:15.593 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:15.593 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.593 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.593 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:15.594 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:18:15.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:15.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:15.596 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 927461a0-cbf0-4f44-ab5d-2bc9be48438c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:15.636 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:15.636 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:15.838 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:15.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.839 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.147667_297.1", 2020-07-17T12:18:15.839 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.839 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-44:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.1", 2020-07-17T12:18:15.839 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.147667727Z", 2020-07-17T12:18:15.839 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.839 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.840 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.840 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KHfS_XyKT5_sDeQiRAnbAOq", 2020-07-17T12:18:15.841 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.841 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.841 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:15.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.842 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.151770_298.1", 2020-07-17T12:18:15.842 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.843 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-44:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.1", 2020-07-17T12:18:15.843 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.151770380Z", 2020-07-17T12:18:15.843 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.843 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.843 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.843 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.844 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.844 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KHfS_XyKT5_sDeQiRAnbAOq", 2020-07-17T12:18:15.844 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.845 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.845 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:15.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.846 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.153820_299.1", 2020-07-17T12:18:15.846 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:15.846 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-44", 2020-07-17T12:18:15.846 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.153820947Z", 2020-07-17T12:18:15.846 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.847 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.847 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.848 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.848 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.849 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zIDj0oMs4SVttfB0Y5UZngn", 2020-07-17T12:18:15.849 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.849 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.850 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:15.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.851 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.157616_300.1", 2020-07-17T12:18:15.851 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:15.851 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-44", 2020-07-17T12:18:15.851 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.157616240Z", 2020-07-17T12:18:15.851 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.852 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.852 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.852 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zIDj0oMs4SVttfB0Y5UZngn", 2020-07-17T12:18:15.853 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.853 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.853 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:15.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.854 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.193545_301.1", 2020-07-17T12:18:15.854 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.854 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-45:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.2", 2020-07-17T12:18:15.855 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.193545729Z", 2020-07-17T12:18:15.855 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.855 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.855 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.855 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.856 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Fsb36ONTqDBQ1OE8f1hFhdu", 2020-07-17T12:18:15.856 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.857 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.857 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:15.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.858 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.197900_302.1", 2020-07-17T12:18:15.858 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:15.858 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-45:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.2", 2020-07-17T12:18:15.858 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.197900832Z", 2020-07-17T12:18:15.859 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.859 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.859 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.859 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.860 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.860 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Fsb36ONTqDBQ1OE8f1hFhdu", 2020-07-17T12:18:15.860 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.860 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.860 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:15.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.861 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.200004_303.1", 2020-07-17T12:18:15.862 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:15.862 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-45", 2020-07-17T12:18:15.862 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.200004581Z", 2020-07-17T12:18:15.862 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.862 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.862 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.863 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.863 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.863 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_32-lN5IG9Yfwq1LuCzvfd1k", 2020-07-17T12:18:15.863 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.864 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.864 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:15.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:15.865 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.203742_304.1", 2020-07-17T12:18:15.865 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:15.865 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-45", 2020-07-17T12:18:15.865 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.203742569Z", 2020-07-17T12:18:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:15.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_32-lN5IG9Yfwq1LuCzvfd1k", 2020-07-17T12:18:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:15.867 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:15.870 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:15.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:15.871 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:15.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:15.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:15.874 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 927461a0-cbf0-4f44-ab5d-2bc9be48438c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:15.913 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:15.913 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:16.119 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:18:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.147667_297.1", 2020-07-17T12:18:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.120 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-44:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.1", 2020-07-17T12:18:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.147667727Z", 2020-07-17T12:18:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.121 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KHfS_XyKT5_sDeQiRAnbAOq", 2020-07-17T12:18:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.151770_298.1", 2020-07-17T12:18:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-44:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.1", 2020-07-17T12:18:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.151770380Z", 2020-07-17T12:18:16.124 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.125 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KHfS_XyKT5_sDeQiRAnbAOq", 2020-07-17T12:18:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:16.126 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.153820_299.1", 2020-07-17T12:18:16.127 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-44", 2020-07-17T12:18:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.153820947Z", 2020-07-17T12:18:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.128 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zIDj0oMs4SVttfB0Y5UZngn", 2020-07-17T12:18:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.157616_300.1", 2020-07-17T12:18:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-44", 2020-07-17T12:18:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.157616240Z", 2020-07-17T12:18:16.131 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.132 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zIDj0oMs4SVttfB0Y5UZngn", 2020-07-17T12:18:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:16.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.193545_301.1", 2020-07-17T12:18:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-45:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.2", 2020-07-17T12:18:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.193545729Z", 2020-07-17T12:18:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.140 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.141 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Fsb36ONTqDBQ1OE8f1hFhdu", 2020-07-17T12:18:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:16.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.197900_302.1", 2020-07-17T12:18:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.143 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-45:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.2", 2020-07-17T12:18:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.197900832Z", 2020-07-17T12:18:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.144 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Fsb36ONTqDBQ1OE8f1hFhdu", 2020-07-17T12:18:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.145 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.200004_303.1", 2020-07-17T12:18:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-45", 2020-07-17T12:18:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.200004581Z", 2020-07-17T12:18:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.147 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.148 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_32-lN5IG9Yfwq1LuCzvfd1k", 2020-07-17T12:18:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:16.149 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.203742_304.1", 2020-07-17T12:18:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-45", 2020-07-17T12:18:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.203742569Z", 2020-07-17T12:18:16.151 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.152 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_32-lN5IG9Yfwq1LuCzvfd1k", 2020-07-17T12:18:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.153 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:16.155 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.156 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:18:16.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:16.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:16.159 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 927461a0-cbf0-4f44-ab5d-2bc9be48438c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:16.199 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:16.200 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:16.384 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:16.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:16.384 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.147667_297.1", 2020-07-17T12:18:16.384 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.385 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-44:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.1", 2020-07-17T12:18:16.385 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.147667727Z", 2020-07-17T12:18:16.385 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:16.385 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:16.385 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:16.386 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:16.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.386 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:16.386 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KHfS_XyKT5_sDeQiRAnbAOq", 2020-07-17T12:18:16.386 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:16.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.387 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:16.387 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:16.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:16.388 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.151770_298.1", 2020-07-17T12:18:16.388 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.388 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-44:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.1", 2020-07-17T12:18:16.388 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.151770380Z", 2020-07-17T12:18:16.388 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:16.389 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:16.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:16.389 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:16.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.389 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:16.390 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_KHfS_XyKT5_sDeQiRAnbAOq", 2020-07-17T12:18:16.390 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:16.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.390 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:16.391 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:16.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:16.391 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.153820_299.1", 2020-07-17T12:18:16.392 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:16.392 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-44", 2020-07-17T12:18:16.392 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.153820947Z", 2020-07-17T12:18:16.392 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zIDj0oMs4SVttfB0Y5UZngn", 2020-07-17T12:18:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.157616_300.1", 2020-07-17T12:18:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-44", 2020-07-17T12:18:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.157616240Z", 2020-07-17T12:18:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_zIDj0oMs4SVttfB0Y5UZngn", 2020-07-17T12:18:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.193545_301.1", 2020-07-17T12:18:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-45:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.2", 2020-07-17T12:18:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.193545729Z", 2020-07-17T12:18:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:16.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.402 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:16.402 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Fsb36ONTqDBQ1OE8f1hFhdu", 2020-07-17T12:18:16.402 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:16.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.403 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:16.403 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:16.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:16.404 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.197900_302.1", 2020-07-17T12:18:16.404 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.404 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-45:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.2", 2020-07-17T12:18:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.197900832Z", 2020-07-17T12:18:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_Fsb36ONTqDBQ1OE8f1hFhdu", 2020-07-17T12:18:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.200004_303.1", 2020-07-17T12:18:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-45", 2020-07-17T12:18:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.200004581Z", 2020-07-17T12:18:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_32-lN5IG9Yfwq1LuCzvfd1k", 2020-07-17T12:18:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "write" 2020-07-17T12:18:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "id": "1_1594988282.203742_304.1", 2020-07-17T12:18:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "section": "bucket", 2020-07-17T12:18:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "lbjidp-45", 2020-07-17T12:18:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:02.203742569Z", 2020-07-17T12:18:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "data": { 2020-07-17T12:18:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "read_version": { 2020-07-17T12:18:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "", 2020-07-17T12:18:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 0 2020-07-17T12:18:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "write_version": { 2020-07-17T12:18:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "_32-lN5IG9Yfwq1LuCzvfd1k", 2020-07-17T12:18:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "ver": 1 2020-07-17T12:18:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "status": { 2020-07-17T12:18:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "complete" 2020-07-17T12:18:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:16.421 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:16.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:16.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:16.422 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 927461a0-cbf0-4f44-ab5d-2bc9be48438c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:16.464 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:16.464 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:16.675 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:18:16.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.676 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.147667_297.1", 2020-07-17T12:18:16.676 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.676 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-44:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.1", 2020-07-17T12:18:16.676 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.147667727Z", 2020-07-17T12:18:16.676 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.677 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.677 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.677 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.677 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.678 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KHfS_XyKT5_sDeQiRAnbAOq", 2020-07-17T12:18:16.678 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.678 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.678 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:16.679 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.679 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.679 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.151770_298.1", 2020-07-17T12:18:16.680 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.680 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-44:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.1", 2020-07-17T12:18:16.680 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.151770380Z", 2020-07-17T12:18:16.680 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.680 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.680 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.681 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.681 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.681 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_KHfS_XyKT5_sDeQiRAnbAOq", 2020-07-17T12:18:16.681 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.683 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.683 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:16.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.684 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.684 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.153820_299.1", 2020-07-17T12:18:16.684 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:16.685 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-44", 2020-07-17T12:18:16.685 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.153820947Z", 2020-07-17T12:18:16.685 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.685 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.685 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.686 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.686 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.686 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zIDj0oMs4SVttfB0Y5UZngn", 2020-07-17T12:18:16.686 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.687 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.688 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:16.689 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.689 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.689 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.157616_300.1", 2020-07-17T12:18:16.690 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:16.690 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-44", 2020-07-17T12:18:16.690 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.157616240Z", 2020-07-17T12:18:16.690 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.690 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.691 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.691 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.691 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.691 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_zIDj0oMs4SVttfB0Y5UZngn", 2020-07-17T12:18:16.692 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.692 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.692 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:16.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.693 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.193545_301.1", 2020-07-17T12:18:16.693 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.694 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-45:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.2", 2020-07-17T12:18:16.694 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.193545729Z", 2020-07-17T12:18:16.694 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.694 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.694 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.695 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.695 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.695 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Fsb36ONTqDBQ1OE8f1hFhdu", 2020-07-17T12:18:16.695 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.696 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.696 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:16.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.697 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.197900_302.1", 2020-07-17T12:18:16.697 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket.instance", 2020-07-17T12:18:16.697 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-45:bc359d1d-af48-401e-ac43-335f84c3f1d9.7028.2", 2020-07-17T12:18:16.697 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.197900832Z", 2020-07-17T12:18:16.697 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.698 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.698 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.698 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.698 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.699 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_Fsb36ONTqDBQ1OE8f1hFhdu", 2020-07-17T12:18:16.699 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.699 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.699 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:16.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.700 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.200004_303.1", 2020-07-17T12:18:16.700 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:16.701 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-45", 2020-07-17T12:18:16.701 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.200004581Z", 2020-07-17T12:18:16.701 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.701 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.701 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.702 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.702 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.702 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_32-lN5IG9Yfwq1LuCzvfd1k", 2020-07-17T12:18:16.702 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.703 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.703 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "write" 2020-07-17T12:18:16.703 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.703 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:16.704 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "1_1594988282.203742_304.1", 2020-07-17T12:18:16.704 INFO:teuthology.orchestra.run.smithi052.stdout: "section": "bucket", 2020-07-17T12:18:16.704 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "lbjidp-45", 2020-07-17T12:18:16.704 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:02.203742569Z", 2020-07-17T12:18:16.704 INFO:teuthology.orchestra.run.smithi052.stdout: "data": { 2020-07-17T12:18:16.705 INFO:teuthology.orchestra.run.smithi052.stdout: "read_version": { 2020-07-17T12:18:16.705 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "", 2020-07-17T12:18:16.705 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 0 2020-07-17T12:18:16.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.705 INFO:teuthology.orchestra.run.smithi052.stdout: "write_version": { 2020-07-17T12:18:16.706 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "_32-lN5IG9Yfwq1LuCzvfd1k", 2020-07-17T12:18:16.706 INFO:teuthology.orchestra.run.smithi052.stdout: "ver": 1 2020-07-17T12:18:16.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:16.706 INFO:teuthology.orchestra.run.smithi052.stdout: "status": { 2020-07-17T12:18:16.713 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "complete" 2020-07-17T12:18:16.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.714 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.714 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:16.714 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:18:16.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:16.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:16.717 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:18:16.753 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:16.753 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:17.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:17.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:17.032 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:18:17.069 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:17.070 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:17.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:17.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:17.496 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:18:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:17.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:17.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:17.775 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:18:17.812 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:17.812 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:18.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:18.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:18.193 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 34954458-202e-48c3-864b-c00b15f41dd5 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:18.232 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:18.232 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:18.462 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:18:18.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:18.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:18.467 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 34954458-202e-48c3-864b-c00b15f41dd5 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:18.506 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:18.506 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:18.754 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2020-07-17T12:18:18.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:18.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:18.758 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 34954458-202e-48c3-864b-c00b15f41dd5 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:18.797 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:18.797 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:19.026 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:18:19.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:19.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '34954458-202e-48c3-864b-c00b15f41dd5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:19.030 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 34954458-202e-48c3-864b-c00b15f41dd5 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:19.076 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:19.076 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:19.320 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2020-07-17T12:18:19.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:19.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:19.323 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 159da1b7-e55f-4be6-b11f-00c3d4a2d701 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:19.597 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:18:19.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:19.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:19.600 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 159da1b7-e55f-4be6-b11f-00c3d4a2d701 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:19.641 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:19.641 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:19.882 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2020-07-17T12:18:19.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:19.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:19.886 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 159da1b7-e55f-4be6-b11f-00c3d4a2d701 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:19.924 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:19.924 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:20.152 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:18:20.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:20.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '159da1b7-e55f-4be6-b11f-00c3d4a2d701', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:20.156 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 159da1b7-e55f-4be6-b11f-00c3d4a2d701 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:20.194 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:20.195 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:20.441 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2020-07-17T12:18:20.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:20.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:20.445 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 927461a0-cbf0-4f44-ab5d-2bc9be48438c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:20.485 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:20.485 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:20.717 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:18:20.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:20.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:20.721 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 927461a0-cbf0-4f44-ab5d-2bc9be48438c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:20.761 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:20.762 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:21.016 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2020-07-17T12:18:21.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:21.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:21.018 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 927461a0-cbf0-4f44-ab5d-2bc9be48438c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:21.057 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:21.058 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:21.267 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:18:21.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:21.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '927461a0-cbf0-4f44-ab5d-2bc9be48438c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:21.271 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 927461a0-cbf0-4f44-ab5d-2bc9be48438c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:21.311 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:21.312 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:21.557 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2020-07-17T12:18:21.560 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-07-17T12:18:21.561 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-46 2020-07-17T12:18:21.601 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=lbjidp-47 2020-07-17T12:18:21.664 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=lbjidp-48 2020-07-17T12:18:21.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:21.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:21.864 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:21.869 INFO:tasks.rgw.c1.client.1.smithi146.stdout:2020-07-17T12:18:21.858+0000 7ff04a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-17T12:18:21.908 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:21.908 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:22.102 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:18:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:18:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988153.112164_495.1", 2020-07-17T12:18:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:15:53.112164Z" 2020-07-17T12:18:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:18:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:18:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988302.013937_407.1", 2020-07-17T12:18:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:18:22.013937Z" 2020-07-17T12:18:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:22.106 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:22.107 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-17T12:18:22.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:22.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:22.109 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:22.152 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:22.378 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:18:22.378 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:18:22.378 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:18:22.379 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:18:22.379 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:18:22.379 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 14149894388601651757 2020-07-17T12:18:22.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:22.379 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:18:22.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:22.380 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:18:22.380 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:22.380 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:22.380 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:18:22.381 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:22.381 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:22.381 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:22.381 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:18:22.381 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:22.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:22.382 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:22.382 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:18:22.383 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:22.383 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:22.383 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988153.112164_495.1", 2020-07-17T12:18:22.383 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:22.383 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:22.384 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:22.384 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.112164Z" 2020-07-17T12:18:22.384 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:22.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:22.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:22.384 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:18:22.385 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:22.385 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:22.385 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:18:22.385 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:22.385 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:22.386 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:22.386 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:18:22.386 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:22.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:22.386 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:22.387 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:18:22.387 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:22.387 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:22.388 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988302.013937_407.1", 2020-07-17T12:18:22.388 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:22.388 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:22.388 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:22.388 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:22.013937Z" 2020-07-17T12:18:22.388 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:22.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:22.389 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:18:22.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:22.389 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:18:22.389 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2020-07-17T12:18:22.390 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2020-07-17T12:18:22.390 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:22.390 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:18:22.392 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-17T12:18:22.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:22.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:22.394 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:22.431 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:22.431 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:22.639 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:22.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.640 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:18:22.640 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:18:22.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.641 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988153.112164_495.1", 2020-07-17T12:18:22.642 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:15:53.112164Z" 2020-07-17T12:18:22.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.642 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:18:22.642 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:18:22.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.643 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988302.598096_409.1", 2020-07-17T12:18:22.643 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:18:22.598096Z" 2020-07-17T12:18:22.643 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:22.644 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:22.645 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-17T12:18:22.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:22.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:22.648 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:22.726 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:22.727 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:22.916 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:18:22.917 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:18:22.917 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:18:22.917 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:18:22.917 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:18:22.918 INFO:teuthology.orchestra.run.smithi146.stdout: "instance_id": 10221300329629434139 2020-07-17T12:18:22.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.918 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:18:22.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.918 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:18:22.919 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:22.919 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:22.919 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:18:22.919 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:22.919 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:22.920 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:22.920 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:18:22.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:22.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.921 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:18:22.921 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:22.921 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:22.921 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988153.112164_495.1", 2020-07-17T12:18:22.921 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:22.922 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:22.922 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:22.922 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.112164Z" 2020-07-17T12:18:22.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:22.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.923 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:18:22.923 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:22.923 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:22.924 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:18:22.924 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:22.924 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:22.924 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:22.924 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:18:22.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:22.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:22.926 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:18:22.926 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:22.926 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:22.927 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988302.598096_409.1", 2020-07-17T12:18:22.927 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:22.927 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:22.927 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:22.927 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:22.598096Z" 2020-07-17T12:18:22.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:22.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:22.928 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:18:22.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:22.929 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:18:22.929 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 0, 2020-07-17T12:18:22.929 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 0 2020-07-17T12:18:22.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:22.930 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:18:22.931 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-17T12:18:22.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:22.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:22.933 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:23.015 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:23.016 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:23.251 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:23.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:23.252 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:18:23.252 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:18:23.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:23.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:23.253 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988153.112164_495.1", 2020-07-17T12:18:23.253 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:15:53.112164Z" 2020-07-17T12:18:23.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:23.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:23.254 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:18:23.254 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:18:23.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:23.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:23.255 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988302.598096_409.1", 2020-07-17T12:18:23.255 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:18:22.598096Z" 2020-07-17T12:18:23.255 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:23.255 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:23.257 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-17T12:18:23.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:23.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:23.259 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:23.304 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:23.304 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:23.677 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:18:23.677 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:18:23.677 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 3210402864628377302 2020-07-17T12:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:18:23.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:23.679 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:18:23.679 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:23.679 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:23.679 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.474476_101.1", 2020-07-17T12:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:22.474476Z" 2020-07-17T12:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:23.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:23.681 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:18:23.681 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:23.681 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:23.681 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988153.112164_495.1", 2020-07-17T12:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:53.112164Z" 2020-07-17T12:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:18:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.852755_90.1", 2020-07-17T12:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.852755Z" 2020-07-17T12:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:18:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988302.598096_409.1", 2020-07-17T12:18:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:22.598096Z" 2020-07-17T12:18:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:23.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:18:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:18:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2020-07-17T12:18:23.687 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2020-07-17T12:18:23.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:23.688 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:18:23.689 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-17T12:18:23.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:23.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:23.690 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:23.768 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:23.769 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:24.006 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:18:24.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:24.006 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.524415_146.1", 2020-07-17T12:18:24.007 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:11:22.524415Z" 2020-07-17T12:18:24.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:24.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:24.007 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988153.162556_482.1", 2020-07-17T12:18:24.007 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:15:53.162556Z" 2020-07-17T12:18:24.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:24.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:24.008 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.617050_107.1", 2020-07-17T12:18:24.008 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:10:10.617050Z" 2020-07-17T12:18:24.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:24.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:24.009 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988302.599482_381.1", 2020-07-17T12:18:24.009 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:18:22.599482Z" 2020-07-17T12:18:24.009 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:24.009 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:18:24.010 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-17T12:18:24.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:24.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:24.012 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:24.054 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:24.054 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:24.289 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:18:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:18:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:18:24.289 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:18:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:18:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "instance_id": 7737347243854991200 2020-07-17T12:18:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:18:24.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:18:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.524415_146.1", 2020-07-17T12:18:24.291 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:22.524415Z" 2020-07-17T12:18:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:18:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988153.162556_482.1", 2020-07-17T12:18:24.293 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.162556Z" 2020-07-17T12:18:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:18:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:24.295 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.617050_107.1", 2020-07-17T12:18:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.617050Z" 2020-07-17T12:18:24.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:18:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:24.297 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988302.599482_381.1", 2020-07-17T12:18:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:22.599482Z" 2020-07-17T12:18:24.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:18:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:18:24.299 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 0, 2020-07-17T12:18:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 0 2020-07-17T12:18:24.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.300 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:18:24.301 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-17T12:18:24.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:24.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:24.303 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:24.342 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:24.342 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:24.585 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2020-07-17T12:18:24.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.524415_146.1", 2020-07-17T12:18:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:11:22.524415Z" 2020-07-17T12:18:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:24.586 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988153.162556_482.1", 2020-07-17T12:18:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:15:53.162556Z" 2020-07-17T12:18:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.617050_107.1", 2020-07-17T12:18:24.587 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:10:10.617050Z" 2020-07-17T12:18:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988302.599482_381.1", 2020-07-17T12:18:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2020-07-17T12:18:22.599482Z" 2020-07-17T12:18:24.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:24.589 INFO:teuthology.orchestra.run.smithi052.stdout:] 2020-07-17T12:18:24.590 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-17T12:18:24.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:24.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:24.591 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:24.634 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:24.634 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:24.869 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:18:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:18:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:18:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:18:24.870 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:18:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "instance_id": 7555103532793312835 2020-07-17T12:18:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:18:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:24.871 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:18:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:24.872 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.524415_146.1", 2020-07-17T12:18:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:22.524415Z" 2020-07-17T12:18:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:18:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:24.879 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988153.162556_482.1", 2020-07-17T12:18:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:53.162556Z" 2020-07-17T12:18:24.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:18:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:24.881 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.617050_107.1", 2020-07-17T12:18:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.617050Z" 2020-07-17T12:18:24.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:18:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:24.883 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988302.599482_381.1", 2020-07-17T12:18:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:24.884 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:22.599482Z" 2020-07-17T12:18:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:18:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:24.885 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:18:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 0, 2020-07-17T12:18:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 0 2020-07-17T12:18:24.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:24.886 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:18:24.887 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-17T12:18:24.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:24.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:24.889 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:24.971 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:24.971 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:25.194 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:25.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.195 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.540719_108.1", 2020-07-17T12:18:25.195 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.540719Z" 2020-07-17T12:18:25.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988152.938812_388.1", 2020-07-17T12:18:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:15:52.938812Z" 2020-07-17T12:18:25.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.197 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.854745_76.1", 2020-07-17T12:18:25.197 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.854745Z" 2020-07-17T12:18:25.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.198 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988302.047780_313.1", 2020-07-17T12:18:25.198 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:18:22.047780Z" 2020-07-17T12:18:25.199 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:25.199 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:25.200 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-17T12:18:25.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:25.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:25.203 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:25.288 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:25.288 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:25.510 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:18:25.510 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:18:25.510 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:18:25.511 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:18:25.511 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:18:25.511 INFO:teuthology.orchestra.run.smithi146.stdout: "instance_id": 3231250023779715025 2020-07-17T12:18:25.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.512 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:18:25.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.512 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:18:25.513 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:25.513 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:25.513 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.540719_108.1", 2020-07-17T12:18:25.513 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:25.513 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:25.514 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:25.514 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:11:22.540719Z" 2020-07-17T12:18:25.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:25.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.515 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:18:25.515 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:25.515 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:25.515 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988152.938812_388.1", 2020-07-17T12:18:25.515 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:25.516 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:25.516 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:25.516 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:15:52.938812Z" 2020-07-17T12:18:25.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:25.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.517 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:18:25.517 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:25.517 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:25.517 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.854745_76.1", 2020-07-17T12:18:25.518 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:25.518 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:25.518 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:25.518 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:10:10.854745Z" 2020-07-17T12:18:25.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:25.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.519 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:18:25.519 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:18:25.519 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "incremental-sync", 2020-07-17T12:18:25.520 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988302.047780_313.1", 2020-07-17T12:18:25.520 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:18:25.520 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:18:25.520 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:18:25.520 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:22.047780Z" 2020-07-17T12:18:25.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:25.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:25.521 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:18:25.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.521 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:18:25.522 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 0, 2020-07-17T12:18:25.522 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 0 2020-07-17T12:18:25.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:25.522 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:18:25.523 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-17T12:18:25.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:25.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:25.525 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:25.563 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:25.564 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:25.792 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:18:25.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.793 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987882.540719_108.1", 2020-07-17T12:18:25.794 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:11:22.540719Z" 2020-07-17T12:18:25.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.794 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988152.938812_388.1", 2020-07-17T12:18:25.795 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:15:52.938812Z" 2020-07-17T12:18:25.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.795 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987810.854745_76.1", 2020-07-17T12:18:25.795 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:10:10.854745Z" 2020-07-17T12:18:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:18:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:18:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988302.047780_313.1", 2020-07-17T12:18:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:18:22.047780Z" 2020-07-17T12:18:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:18:25.797 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:18:25.798 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-17T12:18:25.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:25.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:25.801 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:25.846 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:25.846 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:26.082 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:18:26.082 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:18:26.083 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:18:26.083 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:18:26.083 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:18:26.084 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 9026716065849677085 2020-07-17T12:18:26.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:26.085 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:18:26.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:26.085 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:18:26.086 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:26.086 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:26.086 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987882.540719_108.1", 2020-07-17T12:18:26.087 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:26.087 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:26.087 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:26.088 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:11:22.540719Z" 2020-07-17T12:18:26.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:26.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:26.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:26.089 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:18:26.089 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:26.090 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:26.090 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988152.938812_388.1", 2020-07-17T12:18:26.090 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:26.090 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:26.091 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:26.091 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:15:52.938812Z" 2020-07-17T12:18:26.091 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:26.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:26.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:26.092 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:18:26.093 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:26.093 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:26.093 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987810.854745_76.1", 2020-07-17T12:18:26.094 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:26.094 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:26.095 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:26.095 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:10:10.854745Z" 2020-07-17T12:18:26.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:26.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:26.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:26.096 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:18:26.097 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:26.097 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2020-07-17T12:18:26.097 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988302.047780_313.1", 2020-07-17T12:18:26.098 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:18:26.098 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:18:26.098 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:18:26.099 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:22.047780Z" 2020-07-17T12:18:26.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:26.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:26.100 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:18:26.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:26.100 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:18:26.101 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2020-07-17T12:18:26.101 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2020-07-17T12:18:26.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:26.102 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:18:26.107 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-17T12:18:26.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:26.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:26.109 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:18:26.146 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:26.146 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:26.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:26.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:26.393 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:26.470 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:26.470 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:26.719 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:18:26.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:26.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:26.721 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:18:26.758 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:26.758 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:27.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:27.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:27.005 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:27.080 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:27.081 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:27.323 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2020-07-17T12:18:27.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:27.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:27.326 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-17T12:18:27.364 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:27.364 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:27.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:27.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:18:27.614 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:18:27.690 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:27.690 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:27.924 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2020-07-17T12:18:27.927 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-07-17T12:18:27.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:27.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:27.929 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:18:27.969 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:27.970 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:27.992 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:18:27.981+0000 7fba3e4490c0 0 failed reading obj info from .rgw.root:zone_info.0c5f4c70-7e48-49ca-abec-dc0f91059509: (2) No such file or directory 2020-07-17T12:18:27.992 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:18:27.981+0000 7fba3e4490c0 0 WARNING: could not read zone params for zone id=0c5f4c70-7e48-49ca-abec-dc0f91059509 name=test-zone3 2020-07-17T12:18:27.993 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:18:27.981+0000 7fba3e4490c0 0 failed reading obj info from .rgw.root:zone_info.bc359d1d-af48-401e-ac43-335f84c3f1d9: (2) No such file or directory 2020-07-17T12:18:27.993 INFO:teuthology.orchestra.run.smithi052.stderr:2020-07-17T12:18:27.981+0000 7fba3e4490c0 0 WARNING: could not read zone params for zone id=bc359d1d-af48-401e-ac43-335f84c3f1d9 name=test-zone1 2020-07-17T12:18:28.004 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:18:28.004 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:18:28.004 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2020-07-17T12:18:28.005 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-07-17T12:18:28.005 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone2.rgw.control", 2020-07-17T12:18:28.005 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-07-17T12:18:28.005 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-07-17T12:18:28.005 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone2.rgw.log", 2020-07-17T12:18:28.006 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-07-17T12:18:28.006 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-07-17T12:18:28.006 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-07-17T12:18:28.006 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-07-17T12:18:28.006 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-07-17T12:18:28.007 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-07-17T12:18:28.007 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-07-17T12:18:28.007 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-07-17T12:18:28.007 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-07-17T12:18:28.007 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2020-07-17T12:18:28.008 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "1R9V3H5FL4JONCEX", 2020-07-17T12:18:28.008 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "LcR3LFWGav9hRFziuKCRCpXMbs1dgJln" 2020-07-17T12:18:28.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.008 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2020-07-17T12:18:28.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.008 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2020-07-17T12:18:28.009 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:18:28.009 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-07-17T12:18:28.009 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2020-07-17T12:18:28.009 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2020-07-17T12:18:28.009 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-07-17T12:18:28.010 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2020-07-17T12:18:28.010 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:28.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.010 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-07-17T12:18:28.010 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2020-07-17T12:18:28.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:28.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:28.011 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.011 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca" 2020-07-17T12:18:28.011 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:18:28.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:28.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-17T12:18:28.014 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-17T12:18:28.097 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:18:28.097 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:18:28.341 INFO:teuthology.orchestra.run.smithi052.stderr:Sending period to new master zone bc359d1d-af48-401e-ac43-335f84c3f1d9 2020-07-17T12:18:28.416 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:18:28.416 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:18:28.416 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 2, 2020-07-17T12:18:28.416 INFO:teuthology.orchestra.run.smithi052.stdout: "predecessor_uuid": "159da1b7-e55f-4be6-b11f-00c3d4a2d701", 2020-07-17T12:18:28.417 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": [ 2020-07-17T12:18:28.417 INFO:teuthology.orchestra.run.smithi052.stdout: "1_1594988221.023402_317.1", 2020-07-17T12:18:28.417 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2020-07-17T12:18:28.417 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2020-07-17T12:18:28.417 INFO:teuthology.orchestra.run.smithi052.stdout: "" 2020-07-17T12:18:28.418 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.418 INFO:teuthology.orchestra.run.smithi052.stdout: "period_map": { 2020-07-17T12:18:28.418 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:18:28.418 INFO:teuthology.orchestra.run.smithi052.stdout: "zonegroups": [ 2020-07-17T12:18:28.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.418 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "test-zonegroup", 2020-07-17T12:18:28.419 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zonegroup", 2020-07-17T12:18:28.419 INFO:teuthology.orchestra.run.smithi052.stdout: "api_name": "", 2020-07-17T12:18:28.419 INFO:teuthology.orchestra.run.smithi052.stdout: "is_master": "true", 2020-07-17T12:18:28.419 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:18:28.419 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:18:28.420 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.420 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames": [], 2020-07-17T12:18:28.420 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames_s3website": [], 2020-07-17T12:18:28.420 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:18:28.420 INFO:teuthology.orchestra.run.smithi052.stdout: "zones": [ 2020-07-17T12:18:28.420 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.421 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:18:28.421 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone3", 2020-07-17T12:18:28.421 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:18:28.421 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8001" 2020-07-17T12:18:28.421 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.422 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:18:28.422 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:18:28.422 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:18:28.422 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:18:28.422 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:18:28.423 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:18:28.423 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:18:28.423 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:18:28.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.423 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.424 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:18:28.424 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone4", 2020-07-17T12:18:28.424 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:18:28.424 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8001" 2020-07-17T12:18:28.424 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.424 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:18:28.425 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:18:28.425 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:18:28.425 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:18:28.425 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "pubsub", 2020-07-17T12:18:28.425 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "false", 2020-07-17T12:18:28.426 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [ 2020-07-17T12:18:28.426 INFO:teuthology.orchestra.run.smithi052.stdout: "test-zone1" 2020-07-17T12:18:28.426 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.426 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:18:28.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.427 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:18:28.427 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone1", 2020-07-17T12:18:28.427 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:18:28.427 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi146.front.sepia.ceph.com:8000" 2020-07-17T12:18:28.427 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.428 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:18:28.428 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:18:28.428 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:18:28.428 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:18:28.428 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:18:28.428 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2020-07-17T12:18:28.429 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:18:28.429 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:18:28.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.429 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:18:28.430 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2020-07-17T12:18:28.430 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2020-07-17T12:18:28.430 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2020-07-17T12:18:28.430 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.430 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2020-07-17T12:18:28.431 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2020-07-17T12:18:28.431 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2020-07-17T12:18:28.431 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2020-07-17T12:18:28.431 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2020-07-17T12:18:28.431 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "false", 2020-07-17T12:18:28.431 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2020-07-17T12:18:28.432 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2020-07-17T12:18:28.432 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:28.432 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.432 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_targets": [ 2020-07-17T12:18:28.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.433 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default-placement", 2020-07-17T12:18:28.433 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2020-07-17T12:18:28.433 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": [ 2020-07-17T12:18:28.433 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD" 2020-07-17T12:18:28.433 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:18:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: "default_placement": "default-placement", 2020-07-17T12:18:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:18:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_policy": { 2020-07-17T12:18:28.434 INFO:teuthology.orchestra.run.smithi052.stdout: "groups": [] 2020-07-17T12:18:28.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:28.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:28.435 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2020-07-17T12:18:28.436 INFO:teuthology.orchestra.run.smithi052.stdout: "short_zone_ids": [ 2020-07-17T12:18:28.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.437 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "0c5f4c70-7e48-49ca-abec-dc0f91059509", 2020-07-17T12:18:28.437 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 2627409927 2020-07-17T12:18:28.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.437 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.437 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "7326649b-420d-48b0-8ce4-28b020e4cf14", 2020-07-17T12:18:28.438 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3854808555 2020-07-17T12:18:28.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.438 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.438 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:18:28.438 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 340514820 2020-07-17T12:18:28.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.439 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:18:28.439 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "f2444b35-3e93-4fbd-99ef-52c98595b68f", 2020-07-17T12:18:28.439 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3926357550 2020-07-17T12:18:28.439 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:28.439 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:18:28.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.440 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zonegroup": "test-zonegroup", 2020-07-17T12:18:28.440 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "bc359d1d-af48-401e-ac43-335f84c3f1d9", 2020-07-17T12:18:28.440 INFO:teuthology.orchestra.run.smithi052.stdout: "period_config": { 2020-07-17T12:18:28.440 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_quota": { 2020-07-17T12:18:28.441 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2020-07-17T12:18:28.441 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2020-07-17T12:18:28.441 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2020-07-17T12:18:28.441 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2020-07-17T12:18:28.441 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2020-07-17T12:18:28.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.442 INFO:teuthology.orchestra.run.smithi052.stdout: "user_quota": { 2020-07-17T12:18:28.442 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2020-07-17T12:18:28.442 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2020-07-17T12:18:28.442 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2020-07-17T12:18:28.442 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2020-07-17T12:18:28.442 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2020-07-17T12:18:28.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:18:28.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:18:28.443 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "a52bbfb6-2eb3-4e35-8521-e506ce6cd2ca", 2020-07-17T12:18:28.443 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_name": "test-realm", 2020-07-17T12:18:28.443 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:18:28.444 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:18:28.445 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-07-17T12:19:28.450 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=lbjidp-49 2020-07-17T12:19:28.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:28.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:28.548 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:19:28.590 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:19:28.590 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:19:28.799 INFO:teuthology.orchestra.run.smithi146.stderr:No --period given, using current period=927461a0-cbf0-4f44-ab5d-2bc9be48438c 2020-07-17T12:19:28.838 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2020-07-17T12:19:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:19:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988368.488876_378.1", 2020-07-17T12:19:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "2020-07-17T12:19:28.488876Z" 2020-07-17T12:19:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:19:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:19:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:19:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:19:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:19:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:19:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:19:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:19:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:19:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:19:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:19:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "last_update": "0.000000" 2020-07-17T12:19:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:19:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:] 2020-07-17T12:19:28.842 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-17T12:19:28.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:28.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:28.844 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:19:28.884 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:19:28.884 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:19:29.123 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:19:29.124 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:19:29.124 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:19:29.124 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:19:29.124 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:19:29.125 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:19:29.125 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:29.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:29.125 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:19:29.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:29.126 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:19:29.126 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:29.126 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:29.126 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:19:29.126 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:29.127 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:19:29.127 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:29.127 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:19:29.127 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:29.127 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:29.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:29.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:29.128 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:19:29.128 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:29.129 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:29.129 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:19:29.129 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:29.129 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:29.129 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:29.130 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:19:29.130 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:29.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:29.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:29.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:29.131 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:19:29.131 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:29.131 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:29.132 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:29.132 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:29.132 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:29.132 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:29.132 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:29.132 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:29.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:29.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:29.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:29.133 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:19:29.133 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:29.134 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:29.134 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:29.134 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:29.134 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:29.134 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:29.134 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:29.135 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:29.135 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:29.135 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:29.135 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:19:29.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:29.135 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:19:29.136 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:19:29.136 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:19:29.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:29.136 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:19:29.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:19:34.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:34.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:34.140 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:19:34.188 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:19:34.188 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:19:34.417 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:19:34.417 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:19:34.418 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:19:34.418 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:19:34.418 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:19:34.418 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:19:34.419 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:34.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:34.419 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:19:34.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:34.419 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:19:34.419 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:34.420 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:34.420 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:19:34.420 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:34.420 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:19:34.420 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:34.421 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:19:34.421 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:34.421 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:34.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:34.421 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:34.422 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:19:34.422 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:34.422 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:34.422 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:19:34.422 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:34.422 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:34.423 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:34.423 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:19:34.423 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:34.423 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:34.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:34.424 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:34.424 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:19:34.424 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:34.424 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:34.424 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:34.424 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:34.425 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:34.425 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:34.425 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:34.425 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:34.425 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:34.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:34.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:34.426 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:19:34.426 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:34.426 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:34.426 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:34.427 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:34.427 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:34.427 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:34.427 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:34.427 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:34.428 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:34.428 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:34.428 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:19:34.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:34.428 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:19:34.429 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:19:34.429 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:19:34.429 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:34.429 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:19:34.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:19:39.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:39.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:39.434 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:19:39.476 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:19:39.477 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:19:39.724 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:19:39.724 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:19:39.724 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:19:39.725 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:19:39.725 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:19:39.725 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:19:39.725 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:39.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:39.726 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:19:39.726 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:39.726 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:19:39.726 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:39.726 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:39.727 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:19:39.727 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:39.727 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:19:39.727 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:39.728 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:19:39.728 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:39.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:39.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:39.729 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:39.729 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:19:39.729 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:39.729 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:39.729 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:19:39.730 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:39.730 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:39.730 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:39.730 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:19:39.730 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:19:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:39.732 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:39.732 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:39.732 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:39.732 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:39.732 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:39.733 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:39.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:39.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:39.733 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:19:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:39.736 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:19:39.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:39.736 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:19:39.736 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:19:39.736 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:19:39.737 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:39.737 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:19:39.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:19:44.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:44.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:44.742 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:19:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:19:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:19:45.043 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:19:45.043 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:19:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:19:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:19:45.044 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:19:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:19:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:45.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:19:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:45.046 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:19:45.047 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:45.047 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:45.048 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:19:45.048 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:45.048 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:19:45.049 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:45.049 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:19:45.049 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:19:45.050 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:45.051 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:45.051 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:19:45.051 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:45.052 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:45.052 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:45.052 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:19:45.053 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:45.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:45.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:45.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:45.054 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:19:45.054 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:45.055 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:45.056 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:45.056 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:45.057 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:45.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:45.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:45.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:45.059 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:19:45.059 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:45.060 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:45.061 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:45.061 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:45.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:45.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:45.062 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:19:45.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:45.062 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:19:45.063 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:19:45.063 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:19:45.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:45.063 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:19:45.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:19:50.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:50.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:50.068 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:19:50.117 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:19:50.117 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:19:50.349 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:19:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:19:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:19:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:19:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:19:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:19:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:19:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:50.351 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:19:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:19:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:19:50.352 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:19:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:50.353 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:50.354 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:19:50.354 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:50.354 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:19:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:19:50.355 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:19:50.356 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:50.357 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:50.358 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:50.358 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:50.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:50.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:50.358 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:19:50.359 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:50.359 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:50.359 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:50.359 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:50.359 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:50.359 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:50.360 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:50.360 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:50.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:50.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:50.361 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:19:50.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:50.361 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:19:50.362 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:19:50.362 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:19:50.362 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:50.362 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:19:50.363 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:19:55.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:55.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:19:55.367 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:19:55.409 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:19:55.410 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:19:55.656 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:19:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:19:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:19:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:19:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:19:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:19:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:19:55.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:55.658 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:19:55.658 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:55.659 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:55.660 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:19:55.660 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:55.660 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:19:55.661 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:55.661 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:19:55.661 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:19:55.662 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:55.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:55.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:55.662 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:19:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:19:55.664 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:55.664 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:55.664 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:55.665 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:19:55.665 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:55.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:55.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:55.666 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:55.666 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:19:55.667 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:55.667 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:55.667 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:55.668 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:55.668 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:55.668 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:55.669 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:55.669 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:55.669 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:55.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:55.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:19:55.670 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:19:55.670 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:19:55.671 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:19:55.671 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:19:55.671 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:19:55.671 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:19:55.672 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:19:55.672 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:19:55.672 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:19:55.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:55.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:55.673 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:19:55.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:19:55.674 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:19:55.674 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:19:55.675 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:19:55.675 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:19:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:19:55.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:19:58.541 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2020-07-17T12:19:58.528+0000 7f79b2fed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-17T12:20:00.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:00.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:00.681 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:20:00.724 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:20:00.724 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:20:00.972 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2020-07-17T12:20:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2020-07-17T12:20:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2020-07-17T12:20:00.972 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2020-07-17T12:20:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2020-07-17T12:20:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:20:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:20:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:20:00.973 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2020-07-17T12:20:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:20:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2020-07-17T12:20:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:20:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:20:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594988368.488876_378.1", 2020-07-17T12:20:00.974 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:20:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2020-07-17T12:20:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:20:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:19:28.488876Z", 2020-07-17T12:20:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 4 2020-07-17T12:20:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:20:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:20:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:20:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2020-07-17T12:20:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:20:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:20:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:20:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:20:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:20:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:20:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:20:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:20:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:20:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:20:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:20:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2020-07-17T12:20:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:20:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:20:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:20:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:20:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:20:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:20:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:20:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:20:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:20:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:20:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2020-07-17T12:20:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2020-07-17T12:20:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2020-07-17T12:20:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2020-07-17T12:20:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2020-07-17T12:20:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2020-07-17T12:20:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2020-07-17T12:20:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2020-07-17T12:20:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2020-07-17T12:20:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2020-07-17T12:20:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:20:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:20:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2020-07-17T12:20:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2020-07-17T12:20:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2020-07-17T12:20:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2020-07-17T12:20:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2020-07-17T12:20:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2020-07-17T12:20:00.983 INFO:teuthology.orchestra.run.smithi052.stdout:} 2020-07-17T12:20:00.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-17T12:20:00.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-17T12:20:00.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:00.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:00.987 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:20:01.028 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:20:01.028 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:20:01.263 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:20:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:20:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:20:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:20:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:20:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:20:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:20:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:20:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:20:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:20:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:20:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:01.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:01.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:01.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:01.272 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:20:01.273 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:01.273 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:01.273 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:20:01.273 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:01.274 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:01.274 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:01.274 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:20:01.275 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:01.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:01.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:01.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:01.276 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:20:01.276 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:01.276 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:01.277 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:01.277 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:01.277 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:01.277 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:01.278 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:01.278 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:01.278 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:01.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:01.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:01.279 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:20:01.279 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:01.280 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:01.280 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:01.280 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:01.281 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:01.281 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:01.281 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:01.281 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:01.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:01.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:01.282 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:20:01.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:01.283 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:20:01.283 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:20:01.284 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:20:01.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:01.284 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:20:01.285 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:20:06.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:06.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:06.290 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:20:06.333 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:20:06.334 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:20:06.560 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:20:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:20:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:20:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:20:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:20:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:20:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:20:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:20:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:20:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:20:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:20:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:20:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:20:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:20:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:20:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:20:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:20:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:20:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:20:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:20:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:06.572 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:20:06.574 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:20:11.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:11.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:11.577 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:20:11.617 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:20:11.617 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:20:11.854 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:20:11.854 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:20:11.855 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:20:11.855 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:20:11.855 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:20:11.855 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:20:11.855 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:11.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:11.856 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:20:11.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:11.856 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:20:11.857 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:11.857 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:11.857 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:20:11.857 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:11.858 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:20:11.858 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:11.858 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:20:11.858 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:11.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:11.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:11.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:11.859 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:20:11.860 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:11.860 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:11.860 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:20:11.861 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:11.861 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:11.861 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:11.861 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:20:11.862 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:11.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:11.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:11.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:11.863 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:20:11.864 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:11.864 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:11.864 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:11.864 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:11.864 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:11.865 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:11.865 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:11.865 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:11.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:11.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:11.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:11.882 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:20:11.882 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:11.882 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:11.882 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:11.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:20:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:20:11.884 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:20:11.885 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:20:11.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:11.885 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:20:11.889 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:20:16.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:16.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:16.897 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:20:17.167 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:20:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:20:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:20:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:20:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:20:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:20:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:20:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:20:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:20:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:20:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:20:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:20:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:20:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:20:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:20:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:20:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:20:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:20:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:20:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:20:17.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:17.181 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:20:17.182 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:20:22.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:22.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:22.184 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:20:22.226 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:20:22.226 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:20:22.449 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:20:22.450 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:20:22.450 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:20:22.450 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:20:22.450 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:20:22.451 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:20:22.451 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:22.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:22.451 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:20:22.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:22.452 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:20:22.452 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:22.452 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:22.452 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:20:22.452 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:22.453 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:20:22.453 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:22.453 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:20:22.453 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:22.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:22.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:22.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:22.454 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:20:22.454 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:22.454 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:22.454 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:20:22.455 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:22.455 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:22.455 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:22.455 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:20:22.455 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:22.455 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:22.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:22.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:22.456 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:20:22.456 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:22.456 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:22.457 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:22.457 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:22.457 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:22.457 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:22.457 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:22.458 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:22.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:22.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:22.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:22.458 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:20:22.458 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:22.459 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:22.459 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:22.459 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:22.459 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:22.459 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:22.460 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:22.460 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:22.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:22.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:22.460 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:20:22.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:22.461 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:20:22.461 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:20:22.461 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:20:22.461 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:22.461 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:20:22.462 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:20:27.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:27.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:27.466 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:20:27.507 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:20:27.507 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:20:27.734 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:20:27.734 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:20:27.734 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:20:27.734 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:20:27.735 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:20:27.735 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:20:27.735 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:27.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:27.735 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:20:27.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:27.736 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:20:27.736 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:27.736 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:27.736 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988301.695210_347.1", 2020-07-17T12:20:27.736 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:27.737 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:20:27.737 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:27.737 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:18:21.695210Z", 2020-07-17T12:20:27.737 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:27.737 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:27.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:27.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:27.739 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:20:27.739 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:27.739 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:27.739 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:20:27.740 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:27.740 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:27.740 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:27.740 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:20:27.740 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:27.741 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:27.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:27.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:27.741 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:20:27.741 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:27.742 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:27.742 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:27.742 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:27.742 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:27.742 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:27.743 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:27.743 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:27.743 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:27.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:27.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:27.744 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:20:27.744 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:27.744 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:27.744 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:27.744 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:27.744 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:27.745 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:27.745 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:27.745 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:27.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:27.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:27.746 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:20:27.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:27.746 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:20:27.746 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:20:27.746 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:20:27.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:27.747 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:20:27.748 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1594988368.488876_378.1 target=1_1594988301.695210_347.1 2020-07-17T12:20:28.536 INFO:tasks.rgw.c1.client.1.smithi146.stdout:2020-07-17T12:20:28.528+0000 7ff049ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-17T12:20:32.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:32.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:32.751 INFO:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:20:32.791 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:20:32.791 INFO:teuthology.orchestra.run.smithi146.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T12:20:33.022 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2020-07-17T12:20:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "sync_status": { 2020-07-17T12:20:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "info": { 2020-07-17T12:20:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "sync", 2020-07-17T12:20:33.023 INFO:teuthology.orchestra.run.smithi146.stdout: "num_shards": 4, 2020-07-17T12:20:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "period": "927461a0-cbf0-4f44-ab5d-2bc9be48438c", 2020-07-17T12:20:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: "markers": [ 2020-07-17T12:20:33.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 0, 2020-07-17T12:20:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594988368.488876_378.1", 2020-07-17T12:20:33.025 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 1, 2020-07-17T12:20:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:19:28.488876Z", 2020-07-17T12:20:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 4 2020-07-17T12:20:33.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 1, 2020-07-17T12:20:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:33.027 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "1_1594987603.895710_86.1", 2020-07-17T12:20:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "2020-07-17T12:06:43.895710Z", 2020-07-17T12:20:33.028 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 2, 2020-07-17T12:20:33.029 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:33.030 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2020-07-17T12:20:33.031 INFO:teuthology.orchestra.run.smithi146.stdout: "key": 3, 2020-07-17T12:20:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "val": { 2020-07-17T12:20:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "state": 1, 2020-07-17T12:20:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "marker": "", 2020-07-17T12:20:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "next_step_marker": "", 2020-07-17T12:20:33.032 INFO:teuthology.orchestra.run.smithi146.stdout: "total_entries": 0, 2020-07-17T12:20:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "pos": 0, 2020-07-17T12:20:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "timestamp": "0.000000", 2020-07-17T12:20:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: "realm_epoch": 2 2020-07-17T12:20:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:33.033 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2020-07-17T12:20:33.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2020-07-17T12:20:33.035 INFO:teuthology.orchestra.run.smithi146.stdout: "full_sync": { 2020-07-17T12:20:33.035 INFO:teuthology.orchestra.run.smithi146.stdout: "total": 1, 2020-07-17T12:20:33.036 INFO:teuthology.orchestra.run.smithi146.stdout: "complete": 1 2020-07-17T12:20:33.036 INFO:teuthology.orchestra.run.smithi146.stdout: } 2020-07-17T12:20:33.036 INFO:teuthology.orchestra.run.smithi146.stdout:} 2020-07-17T12:20:33.037 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-17T12:20:33.038 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-17T12:20:33.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:33.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-17T12:20:33.040 INFO:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-17T12:20:33.079 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2020-07-17T12:20:33.080 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2020-07-17T23:41:59.002 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-07-17T23:41:59.036 DEBUG:teuthology.task.console_log:Killing console logger for smithi146 2020-07-17T23:41:59.037 DEBUG:teuthology.task.console_log:Killing console logger for smithi052 2020-07-17T23:41:59.038 DEBUG:teuthology.task.console_log:Killing console logger for smithi146 2020-07-17T23:41:59.038 DEBUG:teuthology.task.console_log:Killing console logger for smithi052 2020-07-17T23:41:59.038 DEBUG:teuthology.exit:Finished running handlers